High-Level Design


Table of Contents

CityNext Initiative

Cities are the hearts and souls of our nations, driving our global economy and impacting our environment. And they are under significant pressure facing challenges like urbanization, modernizing aging infrastructure and sustaining the health and safety of their citizens under economic austerity. They need help which a new era of innovation can provide. But today’s conversation around smart cities needs to broaden beyond infrastructure. Because at their core, cities are about people.

Focusing solely on infrastructure is only limiting the vast potential that is acutely within grasp, and can ultimately be a hindrance to innovation and alleviating the unparalleled issues that many cities are facing today. Microsoft realizes the importance that people play in nation-building, and that is why we are forging this newest endeavor, in order to support and propel city-building.

Microsoft CityNext supports cities to take a people-first approach to innovation that empowers them to change the world now and for the next generation. People-first means harnessing all of the power and intention of people in government, people in business and people who reside in cities to create healthier, greener, safer, more successful places to live. As society continues to evolve and as cities gradually modernize, there are increasing expectations from citizens, workers and businesses to go beyond optimizing operations. The desire for the ability to interact with “one-city”, and for services and information to be disseminated and shared in a “one-stop” way, lends credence to the monumental value that Microsoft CityNext offers to every city.

Microsoft is uniquely equipped to enable this people-first approach because no other company offers the broad portfolio of familiar, secure consumer to business technologies, devices and apps; our vast network of more than 700,000 partners worldwide with relationships in nearly every major city around the globe; and our history of successful education and social programs which, when combined, help cities face what’s next.

Back to top

Profound shift in population mobility through an urban renaissance

The world is undergoing an urban renaissance as more and more people are moving to cities to build better lives. For the first time in history more than 50 percent of the world’s population lives in urban areas. And, by 2050, 76 percent of the global population – 6 billion people – will live in cities. Sixty percent of the world’s GDP is created in cities. And, despite only representing 2 percent of the world's surface area, cities are responsible for 75 percent of the world's energy consumption. They are gathering places for higher education and advanced innovation in healthcare, science and energy – the engines of new ideas and businesses that drive breakthroughs and economic growth.

This ongoing migration creates unprecedented challenges, but even urban areas with slower population growth, face issues that are no less serious. Aging infrastructure, new hazards to public health and safety, and the inherent uncertainty of natural disasters, climate change and global and national economic forces are stark realities cities face. City leaders must meet these growing demands with tight budgets and greater citizen expectations while working across complex, siloed agencies.

City projects are putting the most effort and resources toward making physical infrastructure smart by embedding sensors and upgrading networking capabilities. While this is a critical foundational step, the conversation to date about infrastructure is too narrow. Truly smart cities must also focus on human engagement to empower people in government, businesses and the community to realize the full potential of this new era of innovation to become places where people thrive. A people-first approach means harnessing these capabilities to help students achieve more through a 1:1 learning experience, giving isolated populations access to needed government services, providing elderly high quality healthcare in their homes, supporting entrepreneurs to see their ideas come to fruition faster, getting commuters home sooner, and giving city employees a real-time, one city view.

Microsoft and our global partner ecosystem are uniquely positioned to enable a people-first approach with the end-to-end devices and services capabilities necessary to span everything from the consumer-related demands of citizens to the mission-critical, enterprise demands of city operations. No other company offers the broad portfolio of familiar technologies, devices and apps; the experienced network of thousands of partners worldwide; relationships in nearly every major city around the globe; and the history of successful education and training programs which, combined, help cities face what’s next. Together with our partners, we want to help your city:
  • Transform operations and infrastructure by connecting systems, data and people across departments to make information more accessible and services more affordable.
  • Engage citizens and businesses by enabling real-time communication through any device and apps.
  • Accelerate innovation and opportunity through programs that prepare youth to become the next generation of highly skilled workers, nurture entrepreneurs’ bold ideas and create jobs that help cities compete in the global marketplace.
Through a people-first approach and strategic partnerships, cities can enable self-perpetuating cycles of innovation, opportunity, and progress for years to come. The result? A city that can compete on the world stage as a premier destination for citizens to thrive.

Back to top

New with less

Having worked with nearly every major city around the world, we know that it’s no longer enough to do more with less. We must combine the power of technology with innovative ideas to do NEW with less. Citizens and businesses expect to connect with city services using technology that offers a more contextual and automated experience, and government needs technology to increase efficiency, reduce costs, and foster a more sustainable environment. To address these economic realities, Microsoft and our partners are helping cities work within their means by building on existing investments and incorporating new innovations at their own pace. Ultimately, creating an innovation model that works for today and sustains through tomorrow.

CityNext enables cities to do “New with Less” to operate more efficiently and serve citizens as “One City” in eight critical functions: government administration; public safety and justice; health and social services; education; energy and water; buildings, infrastructure & planning; tourism, recreation, and culture and transportation. Microsoft and our partners are committed to helping citizens, business and government realize this vision through a new era of innovation, including:
  • Cloud – Through the cloud we are connecting public and private data sources with the privacy, security and control needed for a city to have effective cross-departmental collaboration and resource sharing. The scalability and cost-effectiveness of cloud services drive cities’ fiscal responsibilities by reducing costs without cutting essential services.
  • Mobility - Through mobile devices such as sensors, smartphones, and tablets, we are helping cities reach citizens anywhere, on any screen. Citizen-centric apps enable people to directly engage and interact with their city governments for services that make life safer and more convenient. Mobile devices also give employees’ access systems remotely from any location, improving productivity and responsiveness.
  • Social - Through social media such as Twitter, Facebook, Skype, and Yammer, we are helping governments have open, two-way communication with citizens to better understand their needs. Cities can also better protect citizens from safety issues through mobile alerts and social channels.
  • Big Data - Through data and analytics, we are helping leaders gain vital insights from millions of devices, social media channels, and business systems to make more accurate decisions, achieve greater efficiency and respond faster in emergency situations.
Cities are already on their journey towards modernizing today. They face many challenges in preparing for what’s ahead – Microsoft CityNext is the bridge to help them get there, now and into the future. Working together, Microsoft, our partners and leaders can make the most of existing investments and find the right combination of solutions, partnerships and social programs needed to accelerate innovation and create cities in which people can make a real impact for a better world.

Back to top

CityNext solution areas

Today, Microsoft and its partners are delivering CityNext solutions that address the most pressing issues. Formidable challenges confront our city leaders today and can negate some of the very elements that make cities attractive in the first place. Through engagement and cooperation, Microsoft is committed to enabling citizens, workers and leaders to overcome those problematic challenges to provide practical optimism for ever-expanding cities.

After more than 20 years of working with city leaders worldwide, we have learned that there are 44 solution areas across eight city domains that can help address 90 percent of the challenges cities face today. The whole is bigger than the sum of its parts, and Microsoft has identified how interlinked each domain is to the forming of a new city solution landscape.
1.3 CityNext Solution Areas.png
Back to top

Transform operations and infrastructure

The reality of transformation through IT often falls short of the promise. Microsoft and our CityNext partners are committed to helping your city fulfill its promise, starting with the understanding that no two cities are the same. We also believe that transformation does not and should not need to start from scratch. It’s an evolution, not a revolution, and new solutions should build upon and work with your city’s existing and potentially diverse investments. CityNext solutions are designed to meet your city’s unique needs.

Enable new capabilities with flexible cloud services. By capitalizing on opportunity of cloud computing, cities can reduce costs, increase efficiency and productivity, and engage people in new and innovative ways. You can realize these benefits while preserving your budget and time for other citizen-focused needs and priorities.
Your employees, citizens and businesses all expect to connect with city services in new ways using technology. Internally, your employees expect to be able to access line-of-business applications and other work content from nearly anywhere, on virtually any device they choose, at any time—whether it’s the middle of the business day or 3 A.M. on a holiday. Externally, your citizens and businesses expect convenient, instantly responsive ways to access city services online. Both internally and externally, users expect their experiences to be smarter, more automated, and more contextual. Microsoft’s cloud services make it possible to deliver the kinds of experiences that today’s users expect without the level of investment and management that would be required with a traditional infrastructure.
Choice and flexibility are key to cloud adoption now and in the future. It should be up to you to decide what services and assets you want to deploy in the cloud and when, so you have the ultimate ability to balance control and flexibility with cost and agility. At Microsoft, we offer a comprehensive range of enterprise cloud offerings that span the continuum of public to private, helping you move to the cloud at your own pace.

Empower city employees with enterprise-ready devices and apps. Giving your employees the ability to stay connected through a mobile device to their department, team members, and constituents from virtually anywhere, at any time, means citizen service does not need to stop the minute they leave the building.
At Microsoft, we recognize that employees will have strong opinions about which device they want to use. Let them choose. We have tailored offerings that make it possible—and easy—to manage and use a mix of devices in your environment, including those that run on Android, Apple, and Windows operating systems. With a diverse range of devices based on the Windows platform, city employees can take advantage of powerful analysis tools and sync case files, reports, models, pictures and settings right from their PCs, tablets, smart phones, and other devices without compromising security, privacy, or compliance.

Innovate on your terms—with modern solutions. To accommodate the dynamic scale and reach of modern-style apps requires a modern platform. CityNext solutions are uniquely positioned to help governments get all the benefits of scale, speed and agility while still protecting existing investments. Our approach can help you take advantage of the cloud on your terms, without fear of lock-in, and with a consistent and comprehensive range of offerings.
Microsoft recognizes that in an increasingly interconnected computing landscape, enabling interoperability between products from different vendors has become more important than ever. Microsoft is committed to establishing and maintaining open connections between our products and non-Microsoft products, and we support relevant industry standards in a way that promotes interoperability. CityNext solutions make it easier for cities to connect systems across a multi-vendor environment, and share data—all in accordance with the distinct security and privacy policies and laws of your particular city. Our goal is to enable you to choose, develop, and support the applications, languages, and tools that are right to run your city today without imposing limitations on your technology options in the future.

Innovate on your terms—with big data. Data is a valuable resource; an asset. There isn’t a city in the world that isn’t swimming in data, but the challenge is to use that data— as well as the wealth of data available from outside the city—in a meaningful way. Cities need solutions that enable data to flow across an infrastructure, from the sensors, meters, machines, systems, devices, social media channels and public data markets, where valuable data is gathered, to the back-end systems where that data can be turned into information and insight that people and machines can act upon.
Microsoft offers the best end-to-end big data solutions to manage any data, of any size, from any source with our industry leading database services. CityNext solutions can deliver a holistic, city-wide view and give employees new intelligence that supports better historical trend and pattern analysis, richer predictive modeling, and more effective real-time decisions, all of which help a city optimize its resources, breakthrough departmental silos and provide better services for less.

Back to top

Engage citizens and businesses

Whether inspired by a pothole in the street, the need for a permit, frustration over the speed of business development, or the desire to contribute, constituents are demanding greater access to and interaction with government. These rising expectations underscore the importance of providing rich, personalized experiences that result in a better-served, more engaged citizens and businesses. Giving citizens and businesses a straightforward way to quickly engage pays off in a variety of ways, including increased visibility into constituents’ needs and positive perceptions of city government.

Deliver personalized services with a citizen-centric approach. Most citizens would prefer to engage with “one” city, not a series of disconnected agencies and processes. With the identity management capabilities of CityNext solutions, people no longer have to repeatedly identify themselves to multiple departments to access records, submit permits, enroll in school, pay their taxes, or share their health data and access care across different hospitals. Instead, they can engage in a streamlined, “one-stop” experience with their city through personalized hubs that maintain and protect their identities.
In addition to facilitating information access, making technology tools themselves more intuitive can improve the experience of citizens and businesses. People already use gestures and speech to interact with their phones, tablets, PCs, and other connected devices. Such natural ways to interact with technologies make it easier to learn how to operate them and, in essence, allow people to further personalize how they engage with their communities via their devices.
Microsoft leads the industry in computer accessibility innovation, which is particularly important for people who experience visual difficulties, pain in the hands or arms, hearing loss, and speech or cognitive challenges. For many citizens who have particular accessibility requirements, the chance to personalize their devices to meet their individual needs not only makes computer use possible, it improves their ability to participate in public discourse and take advantage of city services and educational opportunities.

Promote real-time dialog with citizens via social media. Citizens want to make their voices heard; city governments want to hear them more clearly. Cities can encourage deeper civic engagement and participation by making the most of mobility and social media communications channels. Microsoft and our CityNext partners help cities integrate familiar social tools into your applications, which can become accessible on many different devices so that city employees can easily share ideas and collaborate with citizens and businesses, regardless of location. Fostering a connected environment gives you the opportunity to improve the value of civic relationships; encourage open dialog; and address people’s perceptions, needs, and civic trends with better speed and understanding.

Increase transparency with open data. Opening up public data – making it accessible to the people - not only increases transparency but also can create opportunities for the development of new services. Offering appropriate, controlled access to previously unavailable data can spur enterprising developers to boost local economic activity and enhance the quality of life for a city’s residents and visitors with useful new solutions and applications.

Back to top

Accelerate innovation and opportunity

Innovation is an essential catalyst for economic growth and competitiveness. However, fully harnessing the power of innovation for a prosperous, competitive city requires more than technology itself. It requires efficient infrastructure, effective institutions, and the creativity of every city’s most important resource—its human capital. City governments can contribute to the sustained economic and social well-being of their communities by attracting talent, helping more people develop useful skills, and expanding technology access to those who otherwise would not be part of the digital society.

Adopt a broad-based partnership approach. The CityNext initiative helps cities enable self-sustaining cycles of innovation that involve whole institutions and individual citizens alike. Microsoft has long participated in partnerships with governments, non-governmental organizations, and other local stakeholders to enable citizens to realize their full potential. Around the world, these programs have helped millions of city residents shape a brighter future for themselves and their local communities.
Today’s economy both requires and enables new business models and public/private partnerships, which you can use to solve big challenges in cost-effective, creative ways.
In keeping with our long-standing commitment to education and opportunity for all young people, Microsoft set a goal in 2012 of creating opportunities for 300 million youth over the next three years through partnerships with governments, nonprofit organizations, and businesses. Through Microsoft YouthSpark, we bring together a range of new, existing, and enhanced programs to address the opportunity divide that faces young people.

Enable the next generation of innovators and entrepreneurs. Cities can help their local technology innovators and entrepreneurs turn ideas into thriving local small and medium-sized enterprises by introducing them to valuable programs and partnerships. For example, visionary entrepreneurs in your area can benefit from the software, support, visibility, and community that Microsoft provides worldwide—at no charge—through Microsoft BizSpark. Advancing the technical knowledge of a city’s younger citizens is also critical to its long-term economic health and competitiveness. That’s why Microsoft DreamSpark gives free access to Microsoft designer and developer tools for high-school and college students and educators.
We want to partner with you to foster creative research, technology, and software solutions and help you grow a sustainable local software economy. You can connect with government, academic, and industry participants at all levels through our state-of-the-art Microsoft Innovation Centers, which offer access to valuable resources, experts, and tools for collaboration and skills development.

Expand digital inclusion. One of the best ways to ensure that all your citizens have basic technology access and literacy is through creative partnerships such as those built through Microsoft Shape the Future. These partnerships support governments’ efforts to achieve their particular digital-access goals, from promoting education reform to developing a 21st-century workforce. You can also give your citizens access to technology in familiar languages through our Microsoft Local Language Program, which aims to empower individuals to create economic opportunities, build IT skills, enhance education outcomes, and sustain their local language and culture. In addition, we provide an online, multilingual Digital Literacy Curriculum, along with free software and support, which you can use to teach and assess basic computer concepts and skills. These help citizens develop new social and economic opportunities for themselves, their families, and their communities.

Provide 21st-century education and skills training. Many city leaders already emphasize access to education for all students as a cornerstone of long-term growth and competitiveness. Because we believe that education is a right, not a privilege, Microsoft and our CityNext partners provide educational institutions with end-to-end learning solutions, including Office 365 for Education, which we offer to students and faculty at no cost.
Getting technology into the hands of young people is not enough. You need to support transformative learning and help them build the skills they need for the future. Government officials, school leaders, and educators can take advantage of Microsoft Partners in Learning to gain new approaches to teaching and using technology to help students develop 21st-century skills. Your local accredited academic institutions can also draw on our Microsoft IT Academy digital curriculum and certifications to provide their students with the technology skills necessary to acquire technical certification for today’s rapidly evolving workplace.

Back to top

Microsoft CityNext Big Data Solution Accelerator

Microsoft CityNext Big Data Solution Accelerator is middleware to host big data-enabled smart city solutions on Microsoft’s cloud platform. It aims to solve key big data issues for smart city scenarios, including data ingestion, data model, city analytics, services integration and information dissemination.
Microsoft CityNext Big Data Solution Accelerator.png

The Microsoft CityNext Big Data Solution Accelerator offers the following key-value propositions to cities:
  • The Microsoft CityNext Big Data Solution Accelerator is built on a modular architecture that allows cities to build smart city solutions with minimal development and quick deployment. Building and deploying a big data architecture seems daunting to even the best IT organizations, and “doing more with less” is a key imperative for cities in their budget-constrained environment. Using a modular architecture that consists of five key modules, the Microsoft CityNext Big Data Solution Accelerator effectively reduces the complexity involved in building big data solutions and can significantly reduce the investment needed while enabling cities to bring innovative new solutions for their citizens.
  • The Microsoft CityNext Big Data Solution Accelerator provides a common framework that allows data and services to be shared across multiple departments and domains. This helps a city to truly leverage the power of big data to perform cross-domain data analysis and to offer integrated services across the entire set of city functions. The scalability and versatility of the solution accelerator ensures that it can meet the needs of a wide range of city services and handle data in various forms – from operational systems, sensor networks as well as social media. The cross-domain nature of the solution accelerator enables cross-agency collaboration and helps uncover insights that lead to positive changes to city services.
  • The Microsoft CityNext Big Data Solution Accelerator is built on Microsoft technologies and products that have consistently endured the test of the marketplace, offering the performance, reliability, and security that a smart city seeks. Together with our partners, we aim to develop the best-of-breed big data ecosystem with enhanced solutions for city big data, city analytics, and city service integration. Through customer POCs (proofs-of-concept), the solution accelerator has demonstrated how cities around the world can leverage cloud computing and big data technologies to ingest, manage and analyze data, disseminate information, and offer a wide range of big data powered services effectively.
With this solution accelerator, a variety of mechanisms are specifically orchestrated to control the overall flow of data so that the end-to-end devices and services capabilities are functioning at the highest levels of efficiency. The Microsoft CityNext Big Data Solution Accelerator connects the dots of the flow of data between the cloud, mobile devices, social media and big data. This essentially cultivates an environment which ensures that valued and vital information is constantly available to every important contributor within the overall city infrastructure so that the notion of “one city” is not just an idea, but a living reality.

Significance and Importance - The Microsoft CityNext Big Data Solution Accelerator uses various methods during the data collection process to consolidate city data from the eight city domains in order to store it in databases. No longer do city employees have to worry about the troubling and exhausting process of going through multiple channels in order to access city data to make decisions.
Frequently, city employees need to address issues that affect multiple functions of the city in real-time. In order for city-wide data to flow across an infrastructure, a solution accelerator must be put into place so that the data can be easily accessible to various domains. A variety of constraints, as well as administrative complexity, can be crippling and can hinder a city’s government from being transparent and accessible.
By unlocking value from data, Microsoft CityNext formulates a new way to optimize, standardize, and sustain an underlying innovative solution accelerator that all city functions can use. Whether the data is structured or unstructured, in the size of gigabytes to petabytes, or at rest or in motion, the Microsoft CityNext Big Data Solution Accelerator will manage countless amounts of city data and disseminate that data accordingly.
Microsoft CityNext Big Data Solution Accelerator b.png

Data Flow Process - The Microsoft CityNext Big Data Solution Accelerator is designed to ingest large volumes of city data, current and historical, and also outside data generated from citizens through social media. It is made up of five main components: data ingestion, city artifacts management, city analytics, city services management and info dissemination.
Initially, the data is acquired from streaming or batch data through the sensor network within the city infrastructure -whether it be through messages, databases or other relevant city files- from city services and from citizens through various clouds (private, hybrid or MS public) and social media networks. Once the data is retrieved, it is distributed to either push or pull channels and routed to the formatter so that the data can be consolidated into the operational data store, or sent directly to the real-time analytics runtime manger based on urgency. After the data is stored, it is analyzed and processed within the repository portal in order to determine through which service the data should be published for rendering.
By using the CityNext Analytics Repository Portal, a hub is created that allows developers and researchers the freedom to publish, share and discover city analytics modules and algorithms, and to connect these modules to specified CityNext data; as well as CityNext analytics-related datasets with predefined data models, for future analysis and subsequent rendering. City researchers and developers also have the opportunity to train and develop their modules and algorithms using a batch processing engine with connected CityNext predefined models. They can also develop and implement their city analytics services using asserts from the CityNext analytic repository.
According to the parameters set, the incoming data can provide batch analysis using city analytics modules and algorithms by Hadoop for unstructured data, and by a pre-built BI data warehouse for structured data with a batch processing engine. Real-time analysis using built-in CEP modules and algorithms through framework based on city streaming data with a real-time processing engine can also be delivered.
Subsequently, in order for the data to be further distributed, based on the nature of the content, the data is rerouted to either the batch-based or real-time analytics runtime manager. Finally, through the service determined by the analysis, data is disseminated through web coverage, a data portal or notification services. The notification services are primarily rendered via notifications and alerts, while web coverage is released mostly based on browsing and queries.
Back to top

The value of the Microsoft CityNext Big Data Solution Accelerator

2.1 The value of the Microsoft CityNext Big Data Solution Accelerator.png
The necessity and importance of having a big data platform is clear and unmistakably evident: it is the engine that powers the initiative towards building a smart city. Without a big data platform, a city would be severely limited in what level of services it offers to its citizens and how creatively it utilizes the available data to fully unleash the power of big data solutions. By using the Microsoft CityNext Big Data Solution Accelerator to supplement a big data platform, unlocking the power to generate big data solutions can be realized. To better understand how the solution accelerator can enhance a big data platform, a deeper look into the components of the solution accelerator can help provide some clarification.

In order for the data to be used, the Data Ingestion component interacts with external and internal data sources to ensure that diverse data be ingested into the big data solution accelerator. With using the communication channels and data formats that the solution accelerator supports, it also provides a flexible framework allowing 3rd party developers to develop additional channels and data formats according to different cities’ demands.

The main data storage of the solution accelerator can be found within the City Artifacts Management component. This particular component supports data enrichment, discovery and aggregation; which is essential for indexing and retrieval, as well as providing OData response of both metadata and result sets with publisher/subscriber (pub/sub) capabilities.

Another facet of the Microsoft Big Data Solution Accelerator is that it can support both real-time and batch-based data. Besides attaining, ingesting and managing city data, it has another component with the unique ability to analyze the stored data to provide and relay readily available solutions, as well as give developers consistent development experience when analyzing data on top of the Microsoft CityNext Big Data Solution Accelerator for more business insights.

Just as vital to the solution accelerator is City Services Management. Service lifecycle management (CSM) enables city operators to manage the big data empowered services registration and release processes. As a result, this component powers various applications in the city so that a collection of services are able to provide essential runtime brokering capability to route the service requests, search the catalog, and utilize services to speed up application development and allow the sampling of smart city services for the city to use.

The final component is equally as indispensable as to why the Microsoft CityNext Big Data Solution Accelerator is necessary in helping a big data platform create a smart city. With the massive amount of incoming data being retrieved, stored and managed and analyzed, it must subsequently be rendered in some present-day fashion. It is here, where the Information Dissemination component shines just as brightly as the others. Information Dissemination (ID) enables the 3rd party to disseminate data through push and pull channels with multiple types of adapters. It also enables a government to control the data that is disseminated and validates the dissemination request from the 3rd party, ultimately ensuring that the data is disseminated to the intentionally targeted users, via the correct channels, at the right time.

Back to top

How can the solution accelerator assist a big data platform in making a smart city?

2.2 How can the solution accelerator assist a big data platform in making a smart city.png
To be a smart city, a city must have a people-first approach, use modern technology and methods to transform in a new age -which will decisively enable engagement between city and citizen- and accelerate innovation and opportunity. The Microsoft CityNext Big Data Solution Accelerator will lead your city towards this path and guide you on the journey.

How, what kind, and how much city data is retrieved is significant in the makings of a smart city. With the Microsoft CityNext Big Data Solution Accelerator, all ingested data is parsed into a key-value pair and stored into permanent ODS storage with extremely high performance. The solution accelerator supports Web HTTP/HTTPS, MS SQL Database, Websphere MQ, and SMB File Share as the pull channels, and HTTP POST as the push channel. Also, the supported data formats include XML, CSV, JSON, and XLS. This ensures that the type of data acquired is not limited to a small variety of data sources, nor the format in which the data comes in.

The capability of the solution accelerator to handle a variety of data and data models is a key to how expandable and scalable it can be. The Microsoft CityNext Big Data Solution Accelerator manages data using NoSQL storage. This NoSQL storage features optimized entity-attribute-value (EAV) schema for HBase with multi-cluster supported indexing and retrieval. The solution accelerator also includes a data warehouse with pre-defined multi-dimension and tabular dimensioned schema based on the Urban Data Model.

Being restricted to analyzing just batch-based data can limit a city’s effectiveness. The Microsoft CityNext Big Data Solution Accelerator provides analytics runtime job management that supports either real-time Complex Event Processing (CEP) jobs, batch-based Win32 or Hadoop jobs for data analysis. The solution accelerator also provides a PowerShell-based Insight Catalog service to create and manage the analytics jobs. The Data Publication Service (DPS) publishes the results from the analytics jobs, Data Warehouse, and OLAP cubes to other developers for further consumption.

Accessibility to data, service information and analytics plays a huge role as to how a city responds to citizens’ needs, optimize services and offer solutions to predictable issues. The Microsoft CityNext Big Data Solution Accelerator is in touch with current-day accessibility standards by allowing service access usage and data access usage with using Live ID, FB, Yahoo, and Gmail for external access and AD for federated login. The solution accelerator also provides tools for the city to create a service model which describes the interfaces, categorizations, and relationships between data/services that provides advanced discovery capability and business opportunities to the developers.

It is impossible for a city to be fully engaged with its citizens without having open channels that encourage and provide constant communication. The Microsoft CityNext Big Data Solution Accelerator enables third parties to disseminate data through push and pull channels with multiple types of adapters including Signal-R, Windows 8, Windows Phone 8, eMail and Windows 8 Live Tile.

Back to top

Architecture

Below is an illustration of a high-level Microsoft CityNext Big Data Solution Accelerator’s architecture data flow.

2.3 Architecture.png

Components - The Microsoft CityNext Big Data Solution Accelerator has all of its components designed to align with our original vision. Below are the five high-level components and a description of each:
  • Data Ingestion – encapsulates all components that are responsible for ingesting data.
    • Data Source Manager – the central managing component for the whole ingestion process including scheduling, as well as interpreting the DSML file in order to bind channels, adaptors, and the formatter together.
    • Ingestions Channels – a pluggable framework in the solution accelerator that enables developers to build or extend more protocols and authentications for ingestions; which includes both push and pull.
    • Formatter – a pluggable framework in the solution accelerator that enables developers to build or extend more data formatting extensions to interpret proprietary formats.
  • City Artifacts Management – the central pieces in the solution accelerator that manage all data storing, enriching, and accessing.
    • Master Data Manager – the central piece of Artifacts Management that enables manageability of entities, schemas, and enrichment processes.
    • Data Enrichment Processor – a processing engine and framework for developers to build or extend post processes to based object tables in raw column-based storage for data enrichments, including calculating columns.
    • Semi-Structured DW Storage – unstructured based storage with table (schema) view definitions that support SQL-based queries with limited relationships.
    • Entity-Based Storage – column-based storage (EAV) for entity-based data access without sophisticated relational queries.
  • City Analytics – for managing and executing analytics with the data available in CAM store and publishes the result sets to available parities.
    • Insight Catalog – supports the registration and discovery of analytics that are available in the solution accelerator for developers for further use.
    • Complex Event Processor – StreamInsight-based engine with the capabilities of binding data sources from CAM and connected to publication system data exposures.
    • Win32 Batch Analytics Runtime Manager – a managing layer for provisioning and executing Win32 executable analytics batch modules.
    • M/R Analytics Runtime Manager – an on-boarding layer to setup readiness for resources, computation, and module functioning prior to an actual job being provisioned.
    • OLAP Cubes/Data Mining – generic SSAS OLAP cubes and data mining services that enable connectivity with CAM for data sources.
  • City Services Management – the gate to the solution accelerator, providing a central end-point for external connectivity while acting as a broker to back-end service providers.
    • Services Manager – life-cycle management for services starting from registration to activation, as well as expirations, and depreciations.
    • Brokering Services – a service bus architecture that enables brokering a request from external to internal service providers, including authentications and rewrite responses.
  • Information Dissemination – a utility that formulates disseminating data based on designated devices, users, along with scheduling and frequencies.
    • Channel Manager – a management service that binds data providers with dissemination channels and configures its interval and schedule, as well as policies and securities.
    • Dispatcher – a pub/sub architecture that dispatches messages to one or multiple channels in a secure way.
    • Dispatch Channels – a framework for developers to build or extend new channels or protocols, and format them for dissemination.

Roles - The Microsoft CityNext Big Data Solution Accelerator is designed with several roles in mind. All security privileges for reference architecture are configured to match the roles listed below.
  • System Administrators – the main account that has the highest level of privileges and access to the system, including the ability to logon to servers interactively. In order to setup and configure the solution accelerator, this account will be needed.
  • Administrators – responsible for operating and monitoring the solution accelerator through management services, with access to platform components. This account is prohibited from logging onto any servers interactively. In order to operate and monitor the solution accelerator, this account will be needed.
  • Services Accounts – a special set of accounts without the ability to logon interactively, targeting solely through the use of services and software services. All communications (IPC) within the solution accelerator are Windows authenticated, either through users’ tokens or services accounts.
  • Developers – a set of users that have the knowledge of the solution accelerator and will create extensions and additional modules or analytics. Once modules or extensions are created, a System Administrator will be needed to configure or register the new modules to the solution accelerator. Some services like analytics are designed to have self-service registration.
  • End-Users – designed as data consumers (both intranet (AD) or external users) that utilize 3rd party Identity Providers (IdPs) like Yahoo, Live ID, Gmail, Facebook, etc.

Back to top

Technological Innovations

After a vision is created the building begins, and eventually that vision successfully comes to fruition. Many times, while the pieces are being put together and the bricks are being stacked to build that foundation, some things are overlooked during the exhaustive and grueling process. However, what can’t be denied as the Microsoft CityNext Big Data Solution Accelerator was created and constructed are the vast Microsoft innovations made in technology that has essentially secured the validity and significance of the Microsoft CityNext Big Data Solution Accelerator. There are numerous innovations that deserve recognition, and each is vital to the functioning of the solution accelerator in totality, but several are very key to what makes solution accelerator important to building a smart city.

2.4 Technological Innovations.png

Data Ingestion - Attaining a variety of data in an INTELLIGENT way
The traditional ETL process for an Enterprise DW leverages a schema-first approach where the data operator has to know the schema he/she is ingesting and where to ingest it, while performing numerous low-valued transformation jobs to make sure that the ingested data matches how it’s being stored or interpreted in the EDW. Whenever new data needs to be ingested, the same process is initiated again.
When working with a big data platform, a higher level of thinking is necessary to deal with both the volume and variety of ingestions because the schema-first approach just isn’t feasible anymore. As a result, the Microsoft CityNext Big Data Solution Accelerator’s data ingestion component intelligently deals with the “E” of ETL by essentially allowing data operators to focus on where to ingest and how to ingest, rather than focusing on what to ingest. Instead of data being ingested, it will be formatted into key-value pairs leaving the CAM to deal with late-schema during read access.
City Artifacts Management - Accessing a high volume of data in an EFFICIENT way
A big data platform not only has to handle the volume of data being stored but also the variety of data being ingested. Unlike the traditional VLDB - where schema is predefined and accessing tremendous volumes of data- a big data platform tends to store data that does not have good schema definitions from the beginning (unstructured). As time goes by, maturity of the data evolves with sets of schemas being defined and refined. However, to combat this problematic issue, the Microsoft CityNext Big Data Solution Accelerator enables the data operator to define one or more schema entities on top of existing data and applies them in real-time. This is what we call the data-first approach.
City Analytics - Maturing machine learning analytics in an EXPERIMENTAL way
One of the values of the Microsoft CityNext Big Data Solution Accelerator is the ability to apply analytics to the data it has stored. In reality, useful analytics do not come for free. It requires the freedom for developers to go through the try-and-error process and to continually learn as the data enhances. The Microsoft CityNext Big Data Solution Accelerator takes a different approach by enabling developers to experiment with various algorithms in a lab environment. Once the analytics prove to be valid, they are shifted to a production environment through self-service registrations and published securely.
City Services Management - Exposing data-driven services in a POWERFUL way
As the gateway to the solution accelerator, the services management component within the Microsoft CityNext Big Data Solution Accelerator provides a traditional ESB-like architecture to expose discoverable services that were registered in the solution accelerator. Again, unlike a traditional ESB, the Microsoft CityNext Big Data Solution Accelerator’s CSM also allows data to be registered directly, making it so unique that data-driven services can be shared and discovered very easily.
In addition, the gateway is also responsible for authenticating access. Currently, the Microsoft CityNext Big Data Solution Accelerator supports the use of external Identity Providers (IdPs) like Live ID, Facebook, Gmail, Yahoo, etc., as well as Active Directory through federations.
Information Dissemination - Connecting a variety of devices in an EFFECTIVE way
The ability to disseminate information or data to external parties (people and/or devices) is an important aspect of a big data platform. The Microsoft CityNext Big Data Solution Accelerator eliminates the complexity for handling apps, analytics, and/or services by declaring who to notify, where the notification messages should go to, how it will reach the receivers, and when these messages will be sent. Most importantly, the Microsoft CityNext Big Data Solution Accelerator also handles the security policies that are required when disseminating messages to external parties.
Management Studio - Managing the solution accelerator in an AFFLUENT way
All components in the Microsoft CityNext Big Data Solution Accelerator are managed through a set of management services. Operators have the option to either use PowerShell Cmdlets to access these management services or utilizing the Management Studio UX that comes with the solution accelerator as a sample for visualized access.
Back to top

Feature Sets

Here is a list of all the features of the Microsoft CityNext Big Data Solution Accelerator. Features marked in red were released in V0.9, while the others were released in V0.

2.5 Feature Sets.png
2.5 Feature Sets b.png
Back to top

Deployment Architecture

The Microsoft CityNext Big Data Solution Accelerator is designed with high scalability in mind. The diagram below illustrates the scalable sets in each layer of the physical architecture, as well as the external accessible units which are outlined in red and green boxes.
2.6 Deployment Architecture.png

The architecture is designed to be layered with cascade accessibility. The Front-End layer will be accessible by external end-users via a corporate firewall, but the Front-End layer won’t have connectivity with the Data layer since the Application layer is blocking the two. Similarly, an administrator can access a Management UX service via the Infrastructure layer, but he/she will not be able to connect or have access to the Front-End, Application, or Data layers.

Front-End Layer - Contains all servers and services that are accessible to external parties (end-users, push data sources, etc.). This layer is the only layer in the architecture that is publicly accessible.
Application Layer - Contains all application servers and services without any actual data being utilized. This layer will not be accessible externally and will not store any business data.
Data Layer - Contains all business data servers and services. This layer will not be accessible externally.
Infrastructure Layer - Contains all infrastructure services and management services across the other three layers. This layer will be accessible through the internal network for both the administrator and developers.
NOTE: For details on installation for multi-box environment, please refer to the Microsoft CityNext Big Data Solution Accelerator – Configuration and Administration Guide.

Back to top

Management Studio

2.7 Management Studio.png
The Microsoft CityNext Big Data Solution Accelerator provides a visual management interface for both the administrator to perform operations and management tasks, as well as for developers to register analytics and services. The Management Studio UX release is made available as a sample feature within the solution accelerator so that the System Integrator (SI) can use it for actual projects such as creating customizations or extensions -granted that proper authentications and authorizations are provided.

The Management Studio UX utilizes Management Services exposed by each module in the solution accelerator to perform management operations. Management Studio UX is also integrated with the Microsoft System Center Operation Manager to perform monitoring and alerting functionalities.

Back to top

City Scenarios

The scenarios that demonstrate the solution accelerator’s capability:
  • The Situational Awareness Dashboard scenario
    • Information: Various data sources including incidents, weather, locations of hospitals and schools, and tweets are ingested using Data Source Markup Language (DSML), stored in HDP HBase, and also shared via the Dynamic OData Adaptor to turn data into information.
    • Intelligence: Hive data warehouse is built in the HDP Big Data environment, and then OLAP Cubes are created in addition to further define multi-dimensions for emergency incidents and citizen complaints trending analysis, which further transforms information into insights. Social media trend analysis is also done through tweets, and TextFlow technology from MSRA is used for the visualization.
    • Integration: Bing Map, Lync real-time collaboration, and 3rd party traffic camera services are integrated via Service Management to take users from insights to action.
  • The Urban Air (UAir) service
    • Based on the Urban Air (UAir) service developed by MSRA, the Best Air Quality Route service was developed to identify the travel route between any two points in the city with the best overall air quality. The route is identified by applying the air quality estimates produced by UAir and road network data to the path finding algorithm. Service capabilities are demonstrated via both Web and Windows Phone clients. The service was developed to showcase Microsoft CityNext Big Data Solution Accelerator’s capability to support developers who would like to develop city services by applying the analytical results were published. On the practical side, the service offers health-conscious citizens a tool to plan travel routes based on air quality and increases awareness of the air quality challenges as the result of rapid urbanization in China.

Below you will find five features that detail the solution accelerator’s capabilities:
  • City Dashboard - City Dashboard is a Windows Store App that supports cross-domain data sharing and analysis, situational awareness and service integration. It helps the city government make decisions more quickly and accurately, address the most pressing social problems, and drive government performance through benchmarking.
  • Situational Awareness Dashboard - Situational Awareness Dashboard is a Windows Store App that serves as the hub for city emergency management. It provides the city government with an overview of real-time situations, point-in-time situations and cross-domain analyses.
  • Sentiment Analysis Dashboard - Sentiment Analysis Dashboard is a Windows Store App that gauges citizen sentiment through the analysis of citizen complaints, news and social media data. It allows city governments to respond to citizen sentiment changes in a timely manner and perform historical sentiment trend analysis.
  • UAir Service - UAir service provides real-time, fine grained, inferred air quality indexes for any given set of geo-coordinates by factoring in air quality monitoring station readings, weather, traffic, and city points of interest data. Service capabilities are demonstrated via both Web and Windows Phone clients.
  • Best Air Quality Route Service - Consuming the air quality indexes produced by UAir service, Best Air Quality Route service identifies the travel route between any two given points that has the best overall air quality. Service capabilities are demonstrated via both Web and Windows Phone clients.
Back to top

Microsoft CityNext Big Data Solution Accelerator and CityNext Initiative Alignment

Below illustrates how the Microsoft CityNext Big Data Solution Accelerator aligns with the CityNext Initiative through its technologies, scenarios, and values.

3 Microsoft CityNext Big Data Solution Accelerator and CityNext Initiative Alignment.png

The CityNext Initiative, together with the Microsoft CityNext Big Data Solution Accelerator is a prime realization about how two ideas can merge with one another in order to align with the ultimate smart city vision. These two concepts coexist in such a way that they provide a mutually beneficial relationship without limiting the other. Because of the technologies that the Microsoft CityNext Big Data Solution Accelerator is constructed of and utilizes, a city is truly transformed, not just by optimizing the infrastructure, but by allowing city employees to offer modern solutions through the use of cloud service capabilities and enterprise-ready devices and apps.

With scenarios that will arise due to normal, everyday social activities and city functioning, a city will finally be able to holistically engage with its citizen base by increasing transparency, utilizing social media so that honest and practical real-time dialog is a staple interactive regularity and services are delivered based directly on the personal needs of all citizens. As a result of the alignment of both the CityNext Initiative along with the Microsoft CityNext Big Data Solution Accelerator, a new wave is on the horizon. One that will drastically expand the innovative minds of the next generation, as well as provide an opportunity to maximize the use of and access to resources, such as intensive education and skills training, for future smart city solutions and technological development.

Back to top

Appendix A: Product Requirements

The China Innovation Group (CIG) within Microsoft Asia-Pacific R&D (ARD) has developed the Microsoft CityNext Big Data Solution Accelerator with the goal of solving problems of data and application silos and of unleashing the power of data possessed by different city agencies. The Microsoft CityNext Big Data Solution Accelerator is middleware built on top of Microsoft products and open source technologies, offering the capabilities of ingesting, consolidating, storing, enriching, analyzing, rendering and disseminating data in meaningful ways.

Technically speaking, the Microsoft CityNext Big Data Solution Accelerator releases in the form of the Solution Accelerator (aka Pre-build artifacts) that installs on top of one or more Microsoft technologies and 3rd party software pre-owned by the customer(s). The Microsoft CityNext Big Data Solution Accelerator’s objective is to accelerate the design, development, and deployment stages during a Smart Cities solution project that has been granted to Microsoft Consultant Services or Microsoft Certified Partners. This Solution Accelerator provides MCS/MCP a fast track for implementing POCs, facilitating the envisioning and planning phases for gap analysis, and/or make use of one or more pre-build artifacts to accelerate the actual development of the project.

As a Solution Accelerator, the Microsoft CityNext Big Data Solution Accelerator deploys and runs on top of a number of Microsoft products, technologies, online services and/or 3rd party software. However, these products, technologies, online services, and/or 3rd party software do not include the Microsoft CityNext Big Data Solution Accelerator itself, rather it is required that customers purchase it separately or manipulate their existing environment or licenses.

Data Ingestion
Data Ingestion (DI) interacts with external and internal data sources to ensure diverse data can be ingested into the solution accelerator. All ingested data is parsed into key-value pairs and stored into permanent ODS storage with extremely high performance.

The component supports Web HTTP/HTTPS, MS SQL Database, Websphere MQ, and SMB File Share as the pull channels, and HTTP POST as the push channel. The supported data formats include XML, CSV, JSON, and XLS. In addition to the above communication channels and data formats, it also provides a flexible framework allowing 3rd party developers to develop additional channels and data formats according to different cities’ demands.

DI.png
City Artifacts Management
City Artifacts Management (CAM) is the main data storage and supports data enrichment, discovery and aggregation. NoSQL storage features optimized entity-attribute-value (EAV) schema for HBase with multi-cluster supported indexing and retrieval, it also provides dynamic building of OData response of both metadata and result sets with publisher/subscriber (pub/sub) capabilities. CAM also includes the data warehouse with pre-defined, multi-dimensional and tabular dimensional schema based on the Urban Data Model.

City Artifacts Management.png
City Analytics
City Analytics (CA) provides analytics runtime job management that supports either real-time Complex Event Processing (CEP) jobs or batch-based Win32 or Hadoop jobs for data analysis. CA also provides PowerShell-based Insight Catalog service to create and manage the analytics jobs. Data Publication Service (DPS) publishes the results from the jobs, Data Warehouse, and OLAP cubes to other developers for further consumption. These CA services will give developers consistent development experience when analyzing data on top of the Microsoft CityNext Big Data Solution Accelerator for more business insights.

City Analytics.png
City Services Management
Service lifecycle management (CSM) enables the city operators to manage the big data empowered services registration and release processes. The services power various applications in the city. Integrated services are a collection of services providing essential runtime brokering capability to route the service requests, catalog search and utility services to speed up the application development, as well as sample smart city services for the city to use. It supports Live ID, FB, Yahoo, and Gmail for external and AD for federated login, while generate services access usage and data access usage. CSM also provides tools for the city to create a service model which describes the interfaces, categorizations, and relationships between data/services, providing advanced discovery capability and business opportunities to the developers.

City Services Management.png
Information Dissemination
Information Dissemination (ID) enables the 3rd party to disseminate data through push and pull channels with multiple types of adapters including Signal-R, Windows 8, Windows Phone 8, eMail and Windows 8 Live Tile to decrease the development effort by employing Information Dissemination capability provided by the Microsoft CityNext Big Data Solution Accelerator. ID also enables government to control data dissemination and validate the dissemination request from the 3rd party, which also ensures that data is disseminated to the intentionally targeted users, with the correct channels, at the right time. Meanwhile, ID enables the on-demand extendibility of adapters through flexible technical framework, certifying the extensive adaptability of the Microsoft CityNext Big Data Solution Accelerator.

Information dissemination.png
Management Studio
A web-based management user interface targeted for both administrators and operators to manage, configure, and monitor the Microsoft CityNext Big Data Solution Accelerator, rather than using a command-line interface in which each previous module had already been provided. This management interface also integrates with the System Center to provide system/product level management and monitoring.

Management Studio.png
City Scenarios
The Microsoft CityNext Big Data Solution Accelerator comes with four end-to-end features that demonstrate its value and its impact on citizens, workers, and leaders in a city. The four feature samples are:

Situational Awareness Dashboard – a dashboard based Windows Store App that targets city workers to help them better understand what is happening within the city by aggregating various information including citizen complaints (311), air traffic, transportation, weather, and real-time traffic cameras. Through various interactions, workers can better visualize a particular situation happening in the city and understand its insight and make better decisions.
City Dashboard – a dashboard and analytic Windows Store App that provides an interactive view on WDI (World Database Index) data and citizen complaints (311) data. Through “drill down” and “slice-n-dice”, a leader or data analyst can understand how their performance compares to others around the world and can also perform self-service business intelligence with cross-domain data including citizen complaints (311) data.
Citizen Sentiments – a dashboard based Windows Store App that targets city workers so they can truly understand what and how citizens in the city feel, as well as give them the ability to collect feedback or suggestions from citizens. Provides a map-based filtering visual that breaks down citizen complaints and suggestions so workers can stay in-tune with multiple regions and districts. A heat map is provided in order to evaluate the type of complaints or suggestions from citizens to give higher priority to those matters that have a higher precedence. An analysis report is also provided to track and trace unsolved complaints and suggestions to avoid cases being lost through the pipeline. In addition, through this dashboard, a city can revise their current process to truly adapt to the most pressing needs of their citizens.
Best Air Quality Route – a web-based application and a Windows Phone 8 application that provides end-users the ability to select their destination locations on a map, and returns the graphical route that has the best air quality during that intended commute.

City Scenarios.png
Back to top

Appendix B: Distributions

The distributions of the Microsoft CityNext Big Data Solution Accelerator come in two forms:
  1. Internal Release to MCS as Solution Accelerator
  2. Private Release to managed partners as Solution Accelerator
The Microsoft CityNext Big Data Solution Accelerator is designed to release in the form of Solution Accelerator targeting both Microsoft Consultant Services and Microsoft Certified Partners. The form of the release is as follows:

Solution Accelerator binaries – contains all runtime binaries and installation scripts about the solution accelerator that is required to be installed on top of existing products/technologies.
SDK – contains developer guides and samples in the form of source code where developers may refer how to make use of the solution accelerator binaries to create the greatest values for customers.
Engagement Documents – includes slides and documents supporting MCS/MCP in order to pitch and sale the values of the solution accelerator to customers during prospect and deal engagement phases.
Operation Documents – documents targeting both operators and administrators for planning, deploying, configuring, and monitoring a running solution accelerator.
NOTE: All source code provided to either MCS or MCP does not contain any encryption algorithms, but rather call on the solution accelerator’s CryptoAPIs to perform the necessary encryptions. All encryption capability of what the shipment provides are based on Microsoft’s solution accelerator capabilities.

Back to top


Last edited Aug 6, 2014 at 9:31 AM by gheadd, version 10