Microsoft CityNext Big Data Solution Accelerator V1 GA


Table of Contents

Introduction

Microsoft CityNext Big Data Solution Accelerator V1 General Availability release is a great upgrade and enhancement based on the Public Preview release with more powerful and innovative functionality and samples, and is the most important milestone for Microsoft CityNext Big Data Solution Accelerator.

New Features

Data Ingestion

  • A unified data ingestion pipeline was created for pull/push data sources.
  • Additional channel plugins to support email/image/Blob data:
    • Data Enrichment
    • Filter and Extraction
  • Fourteen filter plugins and four enrichment function extensions were added.
  • Additional authentication options for Push Ingestion Service.

City Artifacts Management

  • Supports Blob storage.
  • Transparent data encryption for data store.

City Analytics

  • None

City Services Management

  • Supports dual authentication (Windows and Token based)

Information Dissemination

  • None
Back to top

Enhancements

Data Ingestion

  • Data Ingestion architecture was redesigned with plugin framework to improve the extensibility of the data ingestion pipeline.
  • Enhanced the built-in plugins with more functionality.

City Artifacts Management

  • Changed the Hive architecture from Hive-to-Hbase to Hive-over-Hbase in order to eliminate the delay between the Hive and City Artifacts Management.
  • Integrating index creation for the MDM service (Master Data Management) to enable the CRUD model (Create, Read, Update, Delete) for indexing through Management Studio.

City Analytics

  • None

City Services Management

  • None

Information Dissemination

  • None
Back to top

Fixes

Data Ingestion

  • The length of time it takes to ingest a large file or many small files in a folder has been shortened.
  • The generated DSML file can now be accepted by the channel agent.

City Artifacts Management

  • Performance issue for creating an index was improved (changed from failure full retry and SQL single insert to failure incremental retry and SQL bulk insert)

City Analytics

  • User will no longer be asked to deploy a job when doing analytics.

City Services Management

  • None

Information Dissemination

  • None
Back to top

Known Issues and Problems

Data Ingestion

  • Description: The push channel only supports HTTP and is not extensible.
    • Workaround: None
  • Description: If the data contains multiple null values, data ingestion will be slow because the current version is using the HBase’s rest API.
    • Workaround: None
  • Description: The Data Ingestion pipeline does not store Data Ingestion status information.
    • Workaround: None

City Artifacts Management

  • Description: Does not support the deletion of object schema and related data.
    • Workaround: None
  • Description: CAM index storage does not support scalability.
    • Workaround: None
  • Description: OData only supports version 3 of OData Protocol, and does not support version 4.
    • Workaround: None
  • Description: OData queries do not support "order by" non-indexed columns.
    • Workaround: None
  • Description: CAM does not use the HBase primary index, which will lead to performance issues.
    • Workaround: None
  • Description: Data backup and maintenance encounters difficulties due to an overwhelming amount of data being stored in one table.
    • Workaround: None

City Analytics

  • None

City Services Management

  • Description: DQS queries do not support input parameters.
    • Workaround: None

Information Dissemination

  • None
Back to top

Last edited Nov 24, 2014 at 8:50 AM by gheadd, version 5