o GENERAL FEATURES, ARCHITECTURE & METAMODEL CHANGES:
- METADATA HARVESTING FROM CLOUD DEPLOYED MM/DG servers, with two possible deployment architectures of the MIMB server:
· Local Network: For a Model Bridge Server connected to a local Metadata Management Server via a dedicated open port on the same network where firewall is not an issue. In such case, the Model Bridge server is controlled by the Metadata Management Server in a classic client/server architecture.
· Over the Web (Cloud): For a Model Bridge Server connected to a remote Metadata Management Server over the Web where a secure firewall is in place. In such case, the Model Bridge Server will proactively connect over http on regular basis to a dedicated remote Metadata Management Server for any metadata import/export activities to perform. This is the recommended architecture for Metadata Management Servers hosted on the cloud needing to access local Model Bridge Servers behind a firewall for metadata harvesting purpose.
o DATA STORES (Flat files, RDBMS, Cloud Databases, Big Data / Hadoop):
- DATA CATALOGING OF THE DATA LAKE over the cloud (e.g. Amazon S3), on big data clusters (e.g. Hadoop distributions), or file servers (POSIX file systems)
with high level data lake directory inventory import bridges such as:
· Amazon S3 (Simple Storage Service) import (beta bridge)
· Apache Hadoop Distributed File System (HDFS Java API) import or (WebHDFS REST API) import (beta bridge)
· Posix (Windows/Linux) File System import (beta bridge)
discovering lower level metadata from the database and file import bridges such as:
· Flat File Database (CSV or Excel) import (beta bridge)
· JSON (JavaScript Object Notation) Database Files import
· Apache Avro Database Files (JSON) import
· Apache Parquet Database Files import
- BIG DATA HADOOP HIVE TECHNOLOGIES (Cloudera, Hortonworks, MapR)
· Apache Hadoop Hive Database (HCatalog via JDBC) import
· Apache Hadoop Hive Database (Web Hcatalog) import
· Apache Hadoop Hive Database (Hive Metastore via JDBC) import (beta bridge)
- NO SQL TECHNOLOGIES
· MongoDB Database (JSON) import
· Apache CouchDb Database (JSON) import
· MarkLogic NoSQL Database (for JSON only) import
· Apache Cassandra Database import
· DataStax Enterprise (Cassandra) import
- SQL New Databases
· Actian Vector (Vectorwise) Database Database (via JDBC) import
· HP Vertica Database (via JDBC) import
· Oracle MySQL Database (via JDBC) import
- SQL Stored Procedures and Functions (major SQL parsing improvements for lineage)
· Microsoft SQL Server Database (via JDBC) import
· Oracle Database (via JDBC) import
· Pivotal GreenPlum Database (via JDBC) import
· PostgreSQL Database (via JDBC) import
· Teradata Database (via JDBC) import
- SQL Scripts (major SQL parsing improvements for lineage)
· Apache (Cloudera, Hortonworks, MapR) Hadoop HiveQL Script import
· IBM DB2 Universal Database (UDB) SQL Script import (beta bridge)
· Oracle Database SQL Script (PL/SQL) import
· Teradata Database SQL Script (BTEQ) import
- SQL DDL import major improvements (instead of live JDBC connections) which is better for security reasons and for cloud architectures
· Apache (Cloudera, Hortonworks, MapR) Hadoop HiveQL DDL import
· Oracle Database DDL import
· Teradata Database SQL DDL import
· Other Database SQL DDL import (beta bridge)
o DATA INTEGRATION (DI/ETL/ELT):
- New Bridges:
· Alteryx import (beta bridge)
- Improved Bridges:
· Informatica PowerCenter import bridge major improvements in performance and memory usage.
· Microsoft SQL Server Integration Services (SSIS) new support for version 2016.
· Talend DI import bridge is now file based instead of API based, includes major improvements for big data, and new support for MDM.
o BUSINESS INTELLIGENCE (BI/OLAP):
- New Bridges:
· IBM Cognos 11 import (for existing Cognos 10 features, plus beta bridge import of the new Cognos 11 Analytics features of Dynamic Cubes and Data Modules)
· Qlik Sense import
· SAP BusinessObjects 4.2 import (for existing BusinessObjects 4.1 features, however new 4.2 only features are not supported yet)
- Improved Bridges:
· Microstrategy import scalability and performance improvements
· Tableau import bridge major improvement in (1) reducing the number of Tableau Data store connections (Upper/Lower case mix, Default vs specified port, database to various schemas), and (2) parsing the Tableau Init SQL (In Tableau data prep / data integration).
o DATA MODELING (DM) and Object Modeling:
- New Bridges:
· erwin Data Modeler 9.7 import/export bridges
· IDERA ER/Studio 2016 import/export bridges
· OMG UML XMI 2.x new export bridge to NoMagic MagicDraw and Sparx Enterprise Architect
· NoMagic MagicDraw new import bridge
- Improved Bridges:
· Minor improvements in all data modeling bridges
o METADATA MANAGEMENT (MM) & DATA GOVERNANCE (DG):
- New Bridges:
· Apache Atlas (for HortonWorks) import
· Cloudera Navigator import (beta bridge)
· IBM Information Governance Catalog (Open IGC) export (Vendor Internal Bridge)