Management Studio User Guide


Table of Contents

Introduction

Management Studio is a developing tool built in addition to the CityNext Big Data Solution Accelerator. It is a mechanism that provides developers the ability to manage big data and the schemas for the solution accelerator.


About

This guide will walk you through everything you need to know about Management Studio. Whether you read this from beginning to end, or jump all around, we hope you will find this guide useful as you get to know Management Studio.


How to Utilize Management Studio

You need to log in before you can use Management Studio.

2 How to Utilize Management Studio new 1.png

On the home page you can see the navigation bar on the left, and the introduction of each menu item on the right. To access a certain menu item you can either click on the navigation bar and any subsequent listed item that follows, or on the arrow button of any menu item listed in the section to the right.

2 How to Utilize Management Studio.png

Back to top

Data Ingestion

Data Ingestion interacts with external and internal data sources to ensure diverse data be ingested into the big data solution accelerator.

There are two ways to ingest data: pull/push. You can pull your data from a pull source through a pull channel, or you can select a push service to monitor your data and push it to the solution accelerator. Every time you create a pull/push source in data ingestion, there will be a related object schema generated automatically with the same name as the source.

Pull method

There are seven pull channel types for you to choose from:

3.1.1 Pull method.png

To create and edit pull channels for data ingestion, click “Pull Channel Management” to enter the management page.

3.1.1 Pull method 2.png

To review the details of the pull channel, click Details button icon.png to enter the details page. The channel name, description, channel type, and its parameters should be shown.

3.1.1 Pull method 3.png

To create a new pull channel, click the New button icon.png button on the lower left corner to enter the pull channel creation page.

Enter the channel name, description (optional), and choose a channel type.

3.1.1 Pull method 4.png

According to the type you choose, other details might be shown on the creation page, and you will need to fill them in.

For example, if you want to create a channel named “MQ311Channel”, choose “MQChannelDriver” as the channel type, and then you need to enter the host name, port, MQ channel name, and MQ manager name.

3.1.1 Pull method 5.png

The details which need you to fill in are as follows:
  1. MQChannelDriver: Host Name, Port(Int32), Channel Name, Manager Name
  2. FileChannelDriver: Share Path, User Name, Password
  3. WebChannelDriver: Site Root, Use Default Credential, User Name, Password (optional): the password for authentication, Accept Untrusted Certificates, Time-Out In Seconds
  4. DBChannelDriver: Connection String, Data Provider, available values
  5. EmailmapChannelDriver: Host, Port, SSL protocol, Login Name, Password, Header key of thread ID
  6. NYC Power Outage Crawler: Con Edison power outage service URL, PSEG power outage service URL
  7. NYC Precipitation Crawler: National weather service URL

Fill in the necessary items, and click the Submit button icon.png button on the lower right of the creation page. The pull source will be created.

To edit a certain pull channel, click the Edit button icon.png button on the lower right corner to enter the edit page.

The edit operation is the same as the creation operation, except that the name and the channel type of the existing pull channel cannot be changed.

To delete the current pull channel, click the Delete button icon.png button.

To create, edit and manage pull sources for data ingestion, click “Pull Source Management” to enter the management page.

To review the details of the pull source, click Details button icon.png to enter the details page. The pull source name, description, identity attributes, and information about the pull channel should be shown.

3.1.1 Pull method 6.png

To create a new pull source, click the New button icon.png button on the lower left corner to enter the pull source creation page. Enter the pull source name, and description (optional).

For example, you can create a pull source named “MQ311” using the channel named “MQ311Channel”, which was created previously.

3.1.1 Pull method 7.png

Click Side arrow button icon.png to go to the select channel page, and select from all of the existing pull channels. For different channels you select, they may belong to different channel types. According to different channel types, there are different items you need to fill in.
  1. MQChannelDriver: Queue Name
  2. FileChannelDriver: Paths, Folder Include Filters, Include sub-folders, Folder Exclude Filters
  3. WebChannelDriver: URL list, Sequence, End The Sequence When Last Output Is Empty
  4. DBChannelDriver: SQL Query, Table Name, Batch Size
  5. EmailmapChannelDriver: Folder Path, Subject Filter
  6. NYC Power Outage Crawler: Pull Con Edison power outage information, Pull PSEG power outage information
  7. NYC Precipitation Crawler: Precipitation type, Precipitation counties

For example, “MQ311” uses “MQ311Channel”, which belongs to the pull type “MQChannelDriver” as its channel.

3.1.1 Pull method 8.png

Click Side arrow button icon.png to go to the select stream filter page, click Add button icon.png and select from the existing stream filters, and fill in the necessary items. “MQ311” uses “XslTransformer” as its stream filter.

3.1.1 Pull method 9.png

Some items require more than one entry. You need to fill in the input box on the left of add button icon 2.png, and click add button icon 2.png every time you finish entering an entry. All the entries of this item will be added one by one to the box on the left of Delete button icon 3.png. To delete an entry, click on it and then click Delete button icon 3.png.

Click Side arrow button icon.png to go to the select stream converter page, and select from nine stream filters. The filters and their relative settings are as follows:
  1. BlobConverter: BLOB Content Column Name
  2. JsonJsonPathConverter: Json Paths, Customized Columns, Exclude Extra Columns, Stream Encoding
  3. CsvConverter: Header Start Row, Delimiter, No Fields Enclosed In Quotes, Quotation Mark, Stream Encoding, Batch Size, Halt On Max Tolerant Errors, Max Tolerant Errors, CSV Data Contains Null Value, Constant String Represented as Null, Strong Type Awareness, Strong Type Delimiter
  4. ExcelConverter: Sheet Names, Head Row, Columns, Maximal Allowed Conversion Failures, Start Row, End Row, Start Column, End Column, Shared Columns, Row Number Column
  5. ImageConverter: Extract Advanced Information, BLOB Content Column Name
  6. HtmlXPathConverter: Stream Encoding, Exclude Default Columns, Disable Converting To MSHtml Doc, Page Load Timeout In Seconds, Row Xpaths, Columns
  7. JsonLinqConverter: Linq Query, Stream Encoding
  8. XmlLinqConverter: Linq Query, Stream Encoding
  9. XmlXPathConverter: Flattening Level, Separator Of SubElement, Stream Encoding, Namespaces, Row Xpaths, Columns

“MQ311” uses “XmlXPathConverter” as its stream converter.

3.1.1 Pull method 10.png

Click Side arrow button icon.png to go to the Select datatable filter page, and then select from eight stream filters. The filters and their relative settings are as follows:
  1. ColumnRenamer: Column and NewName Pairs
  2. ComputedColumnFilter: Computed Columns And Rules, Max Tolerant Errors
  3. ColumnFilter: Columns To Filter
  4. TimeStamper: Timestamp Column Name, Timestamp Type, Timestamp String Format
  5. KeywordExtractor: Keywords file path, Case Sensitive, Source column Name, Keywords column name
  6. DataTimeAdjuster: Columns To Adjust
  7. RegexValidator: Column Validation Settings
  8. StrongTypeTransformer: Columns To Transform, Ignore Errors, Set To Default Value On Error

“MQ311” uses “StrongTypeTransformer” as its datatable filter.

3.1.1 Pull method 11.png

At the end of this page, you need to set the identity attributes of the pull source you are creating. Here we have no identity attributes for “MQ311”, so nothing should be added.

3.1.1 Pull method 12.png

Click Side arrow button icon.png to go to the preview page, and check all the items.

3.1.1 Pull method 13.png

To test the pull source before it is officially created, click the Test Run button icon.png button on the lower right corner of some of the pages.

To start immediately after submitting, mark the checkbox at the end of the preview page.

3.1.1 Pull method 14.png

If no errors occur, click the Submit button icon.png button on the lower right of the preview page. To change the settings before submission, click the back arrow to go back to previous steps.

To edit a certain pull source, click the Edit button icon.png button on the lower right corner to enter the edit page.
The edit operation is the same as the creation operation, except that the name of the existing pull source cannot be changed.

To delete the current pull source, click the Delete button icon.png button.

To set a schedule and let the pull source run automatically, click the schedules button icon.png button to enter the schedule page. Then you can create a new schedule, or if there is already at least one schedule, delete the current schedule.

To let the pull source run mandatorily, click the start button icon.png button.

When the current pull source runs automatically or mandatorily, the run status will be shown as “running”. After it has finished running, the status will be shown as “succeeded” or “failed”.

3.1.1 Pull method 15.png

To disable a pull source whose status is “Active”, click the Disable button icon.png button. To enable a pull source whose status is “disabled”, click the Enable button icon.png button (Note: before managing the index for a certain object schema, the related pull source must be disabled; before a certain pull source starts running, it must be enabled).

Back to top

Push method

To create, edit and manage push sources for data ingestion, click “Push Channel Management” to enter the management page. The operation is similar to pull source management, yet even simpler.

3.1.2 Push method.png

To review the details of the push source, use the Details button icon.png button.

To create a new push source, click the New button icon.png button on the lower left corner to enter the push source creation page. Enter the push source name and description, select content type (Stream/Data Table) and authentication (select from None, Windows, Basic) and add the datatable filter and identity attributes.

To edit a certain push source, use the Edit button icon.png button.

To delete the current push source, click the Delete button icon.png button.

To disable a push source whose status is “Active”, click the Disable button icon.png button. To enable a push source whose status is “disabled”, click the Enable button icon.png button (Note: before managing the index for a certain object schema, the related push source must be disabled).

Back to top

City Artifacts Management

City Artifacts Management is the main data storage and supports data enrichment, discovery and aggregation.

Object schema management

On this page, you can review and manage indexes and computed attributes of the object schema, which store and manage the data.

2.2.1 Object schema management.png


To review the object schemas in alphabetical order, click the up/down arrow button beside “name”.

To catch a glimpse of the name, the attributes or computed attributes, move the mouse over the relative item as shown below:

2.2.1 Object schema management 2.png


To review and manage a certain object schema, select it with a click.

2.2.1 Object schema management 3.png


To review the details of the schema, click Details button icon.png to enter the details page. The name, index, raw attributes (name, type, indexed status), and computed attributes (name, type, computation rule) should be shown.

2.2.1 Object schema management 4.png


To review the data of the schema, click Data button icon.png to enter the data page. The values for each attribute, including computed attributes, should be shown.

2.2.1 Object schema management 5.png


To manage the indexes, click Indexes button icon.png to enter the index page. Click Open check box icon.png to create an index for the unindexed attribute. Click Checked box icon.png to delete index for the indexed attribute.

2.2.1 Object schema management 6.png
Note: indexing will occur in the background and may take some time to complete depending on the data size.


Computed attributes are attributes defined by the computed result of some existing attributes.

To manage the computed attributes, click Computed attributes button icon.png on the lower right corner to enter the computed attribute list page.

2.2.1 Object schema management 7.png


To create a new computed attribute, click New button icon.png on the lower left corner to the enter computed attribute creation page.

2.2.1 Object schema management 8.png


Enter the name (Note: the name can only contain letters, numbers and underscores, and starts with letters).

Choose the data type. There are 24 types for you to choose from: sbyte, sbyte?, short, short?, int, int?, long, long?, float, float?, double, double?, bool, bool?, guid, guid?, string, binary, datetime, datetime?, decimal, decimal?, byte, byte?.

2.2.1 Object schema management 9.png


Enter the computation rule. Click Question mark button icon.png to get help with the rule. The help page lists operators and functions you may need to set the rule.

2.2.1 Object schema management 10.png


For example, to create a computed attribute to concat two attributes of string type “Country” and “City”, enter “Concat(Country, City)”. Click Validate button icon.png and then you can get a preview of the computed date (Note: the data will not be saved until you submit the creation).

2.2.1 Object schema management 11.png


Click Submit button icon.png. The computed attribute and the according data will be saved. The new computed attribute should be seen on the computed attribute list page.

To edit a certain computed attribute, click Edit button icon.png on the lower right corner to enter the edit page.

2.2.1 Object schema management 12.png


The edit operation is the same as the creation operation, except that the name of the existing computed attribute cannot be changed.

To delete the current computed attribute, click Delete button icon.png.

Back to top

Entity schema management

On this page you can create and edit entity schema. You can choose the data that interests you from object schema, and map them to some entity schemas, in order to review and manage them better.

Entity schema management.png


To review the entity schemas in alphabetical order, click the up/down arrow button beside “Name”; for chronological order, click the button beside “Updated Time”.

To catch a glimpse of the attributes, the bindings, and the permissions, move the mouse over the related item.

To review and manage a certain entity schema, select it with a click.

Entity schema management 2.png


To review the details of the schema, click Details button icon.png to enter the details page. The name, attributes, binding schema, permission status should be shown.

Cannot resolve image macro, invalid image name or id.


To review the data of the schema, click Data button icon.png to enter the data page.

Entity schema management data page img.png

To create a new computed attribute, click New button icon.png on the lower left corner to enter the entity schema creation page.

Enter the name and choose a creation type.

Entity schema management 4.png


There are three ways to create a new entity:
  1. Create all of the entity attributes and the object schema they are mapped from.
  2. Copy attributes from object schema. You need to choose an existing object schema. All of its attributes will be mapped to an entity attribute.
  3. Copy from entity schema. You need to choose an existing entity schema. All of its attributes will be copied to the new entity schema.

Click Side arrow button icon.png to go to the properties & mappings page. The entity attributes which are mapped from existing object schema or copied from existing entity schema should already be listed.

For example, you can create an entity schema named “nyc311realtime”, and copy attributes from the object schema “MQ311”, which was generated automatically after creating the pull source with the same name.

Entity schema management 5.png

Entity schema management 6.png


If you want to create other entity attributes or other mappings, click Plus sign button icon.png below the entity attributes.

Entity schema management 7.png


Enter the name and choose the data type, and click Floppy disk button icon.png. Choose an attribute for the current object schema ( Note: there are choices only if there is any available property in this object for the corresponding data type).

Entity schema management 8.png

Entity schema management 9.png


Click Plus sign button icon.png on the right to add an object schema for mapping.
Every existing item in this table can be edited or deleted. To edit or delete a certain item, move your mouse over it.

Edit and delete button icon.png


To edit a certain entity schema, click Edit button icon.png on the lower right corner to enter the edit page.

Entity schema management 10.png


To delete the current computed attribute, click Delete button icon.png.

To edit the permission status of the current computed attribute, click Permissions button icon.png to enter the permissions page.

Entity schema management 11.png


Click Add button icon.png to add a new permission.

Entity schema management 12.png


Choose everyone from the drop down box under the User/Group column, enter the account and select “allow” under the permission column in order to grant the user/group access, or “denied” to deny the user/group access.

Entity schema management 13.png


Click Delete button icon 2.png to delete the current permission.

Click Submit button icon.png to submit the edit.

Note: only the user/group with “allow” permission can access the related OData URL.

Back to top

City Services Management

City Services Management enables operators to manage the big data empowered services registration and release.

Business entity management

On this page you can create and edit business entities which use entity schema to gather information about the relative data.

3.3.1 Business Entity Management.png

To review the details of the schema, click Details button icon.png to enter the details page. The name, description, publisher, query type, query statement, and data query URL should all be shown.

3.3.1 Business Entity Management 2.png

To create a new business entity, click New button icon.png button on the lower left corner to enter the business entity creation page.

Enter the name, description (optional), publisher, query statement, and choose a query type. Choose OData to gather information from Odata service, or Hive to gather information from the Hive table.

For example, you can create a new business entity named “311NoHeatNoWaterToday”, which uses the entity schema named “nyc311realtime” to gather information about the relative data.

3.3.1 Business Entity Management 3.png

The query statement can be written as shown below:

3.3.1 Business Entity Management 4.png

Meaning that you can gather information regarding the entity schema “nyc311realtime” (the attributes (createddate, agency, complainttype, descriptor, incidentaddress) with a created date later than today (eastern standard time); an agency that is equal to 'HPD'; a complaint type that equals 'Heat/Hot Water'; and displaying them in descending order according to the created date.

To edit a certain business entity, click the Edit button icon.png button on the lower right corner to enter the edit page.
The edit operation is the same as the creation operation, except that the name of the existing business entity cannot be changed.

To execute the current business entity, click the Execute button icon.png button.

For example, if you execute the entity “311DownedTreeTodayCount”, the browser will ask you whether to open or save 311DownedTreeTodayCount.json(13 bytes) from fearr.cloudapp.net.

3.3.1 Business Entity Management 5.png

The .json file should contain the data that is the direct result of the relative query.

To delete the current business entity, click the Delete button icon.png button.

Back to top

Diagnostics

Functional self-test

On this page you can create and run diagnostics on demand in order to check the functionality of the solution accelerator.

3.4.1 Functional Self-Test.png

To start a new functional self-test, click the New button icon.png button.

To view the result of the current functional self-test, click the View result button icon.png button to enter the result page, and the details should be shown (i.e. which test cases have run, and the results).

Click the View log button icon.png button to view the log. See an example below:

3.4.1 Functional Self-Test 2.png

SCOM report

On this page you can view the SCOM daily health check report.

3.4.2 SCOM Report.png

To view the details of the current SCOM report, click the Details button icon.png button to enter the details page.

Data flow diagram

On this page you can view a data flow diagram, which clearly shows how data sources, objects, entities, and queries are all linked together. There are three types of data flow diagrams: tree view, line view, and force view.


1. A Tree View that spins in a circle with each connection being properly aligned according to the nearest distance from the center of the circle.

3.4.3 Data Flow Diagram.png

Click on a node to focus on related nodes and their relationship; resulting in the connections of the related nodes being shown as a tree.

3.4.3 Data Flow Diagram 2.png



2. A Line View (the default type) that connects the dots. The size of the dots indicate the size of each object.

3.4.3 Data Flow Diagram 3.png

Hang your mouse over a certain node, the related nodes and their relationship should be highlighted. Or you can click on a node to focus on related nodes and their relationship.

3.4.3 Data Flow Diagram 4.png 3.4.3 Data Flow Diagram 5.png



3. A Force View that supports more interactive dragging, while still understanding and displaying the impact that each movement has on the overall relationship between each node.

3.4.3 Data Flow Diagram 6.png

Click on a node to focus on related nodes and their relationship.


3.4.3 Data Flow Diagram 7.png

Also, you can drag a certain node anywhere you like.

Note: To choose the type of data flow diagram, click any of the following buttons Force layout button icon.png Tree layout button icon.png Line layout button icon.png in the bottom right corner.

Back to top

Last edited Jan 21, 2015 at 7:39 AM by gheadd, version 8