o DATA / HIERRACHICAL
STRUCTURES AND ARRAYS (JSON, etc.):
Major improvements of JSON file
modeling (metamodel/profile) including new support for root object types,
arrays, and more.
Consequently any use of JSON in the following import
bridges has been redesigned, and therefore requires a full import (i.e. without
incremental harvesting):
- FILE SYSTEMS (Object Stores, Data Lake, etc) with JSON files inside:
· Apache Hadoop Distributed File System (HDFS Java API)
· Amazon AWS Simple Storage Service (S3) File System
· Microsoft Azure Data Lake Storage Gen2
· Microsoft Azure Blob Storage File System
· Google Cloud Storage (GCS) File System
· Confluent Kafka File System (API and Schema Registry)
· OpenStack Swift Object Store File System
- NoSQL databases based on JSON:
· Apache CouchDB NoSQL Database
- ETL/ELT/DI/BI bridges reading/writing JSON file based data stores, such as the following import bridges:
· Informatica PowerCenter (Repository)
o DATA / COMPLEX TYPES
(Arrays, Structures):
New support for complex types (arrays and
structures) often on tables implemented/externalized as JSON files, on some
import bridges:
- Databases supporting complex types implemented as external files, such as the following import bridges:
· Apache Hadoop Hive Database (HCatalog and Metastore via JDBC)
· Cloudera Data Platform Hadoop Hive Database (HCatalog and Metastore via JDBC)
· Databricks Delta Lake Hadoop Hive Database (HCatalog and Metastore via JDBC)
· Google Cloud Platform (GCP) Hadoop Hive Database (HCatalog and Metastore via JDBC)
· Google BigQuery Database (via JDBC)
· Snowflake Database (via JDBC)
- ETL/ELT/DI/BI import bridges reading/writing database complex types, such as the following import bridges:
· Apache Hadoop Hive Database SQL DML (DI/ETL) Script (HiveQL)
· Informatica PowerCenter (Repository)
o DATA / RELATIONAL DATABASES (RDBMS, Big Data / Hadoop Hive, etc.):
- Amazon Web Services (AWS) Athena Database import bridge: new bridge
- Amazon Web Services (AWS) Aurora Database import bridge: new bridge
- Apache Druid Database import bridge: new bridge
- Google BigQuery Database import bridge: redesigned and improved
- Google Spanner Database (via JDBC) import bridge: new bridge
- Presto Database import bridge: new bridge
- SAP Sybase ASE Database import bridge: redesigned and improved
o DATA / FILE SYSTEMS (Object Stores, Data Lake, Flat Files / CSV, etc.):
- Confluent Kafka Schema Registry import bridge major improvements
- Microsoft Azure Data Lake Storage Gen2 import bridge: major improvements
o DATA / NoSQL DATABASES (JSON, etc.):
- Microsoft Azure Cosmos DB import bridge: new bridge
o DATA INTEGRATION (DI/ETL/ELT/Scripts):
- Metadata Excel Format import bridge (now over 10 years old) does not provide the scalability and capabilities required to support the many new NoSQL and cloud technologies. This bridge is now deprecated in 2022 and will be End Of Life (EOL) unsupported in 2024, as it is replaced by the new Data Mapping Script import bridge below.
-
Data Mapping
Script import bridge:
new import bridge from data mapping SQL like scripts supporting any
source/target technologies (RDBMS, NoSQL, file systems, etc.) to emulate
proprietary (or non supported) DI/ETL/ELT/CDC tools and
technologies.
This new Data Mapping Script import is a much more capable
and scalable evolution and replacement of the 10 years old Metadata Mapping
Excel Format import bridge which
has been deprecated in 2022 and will be End Of Life (EOL) unsupported in
2024.
The new data mapping script format provides a number of advantages
over the older metadata Excel format (deprecated) including:
· The data mapping script format supports a full round trip (export/import) from/to a data mapping, while this is not possible with the older metadata Excel format where the export was more for reporting and the import for bootstrapping.
· The data mapping script format files are generally much more compact than the very verbose metadata Excel format files based on classifier/table maps and feature/column maps.
· The data mapping script format is easier to generate from SQL based DI/ETL/ELT/CDC tools.
· The data mapping script format supports both bulk mappings and query mappings, while the metadata Excel format are quite limited in its handling of bulk mappings.
· The data mapping script format supports more source and target data store technologies (RDBMS, NoSQL, File Systems) that is supported by the metadata Excel format.
- Amazon Web Services (AWS) Glue ETL (via Apache Spark) import bridge: major improvements.
- Databricks Unified Data Services (via Apache Spark) import bridge
- Microsoft Azure Data Factory import bridge: major improvements.
- Informatica PowerCenter import bridge: major improvement for fast incremental harvesting (up to 10x faster) based on PowerCenter Repository XML incremental generation and rebuild.
- Snowflake Database SQL DML (DI/ETL) Script (SnowSQL) import bridge: major improvements to support the ETL aspects like SnowPipe, Stream, Task.
- Talend Data Integration import bridge: major improvements including support for numerous new data connectors and transformations, as well as new support for internal parameter overwrite to improve lineage.
o BUSINESS INTELLIGENCE (BI/OLAP):
-
Microsoft Power BI (PBIX or
PBIT) File (PowerQuery M language based) import bridge,
Microsoft Power BI Report Server
(Repository) import bridge,
and
Microsoft Azure Power BI Service
(Repository) import bridge:
major improvements including new Remote browsing, authentication. However, this
bridge remains limited (compared to other BI server import bridges) because of
Microsoft Power BI's current limited metadata access (API and file formats).
More major improvements are expected in the next release pending Microsoft
progress.
- Microsoft SQL Server Analysis Services (SSAS) XMLA file import bridge: redesigned in java, and new multi dimensional modeling in addition to tabular modeling
- Tableau Hyper (Data Extract) Data Store import bridge: major improvements including support for the latest version and API
o BUSINESS APPLICATION (ERP, etc.):
- SAP Business Warehouse 4 HANA (BW/4HANA) import bridge: improved ABAP scripts, and support for Query Reports
o MIMB CREDENTIALS ON CLOUD
SECRET MANAGERS:
allowing administrators to store bridge credentials (e.g.
user, password and private key) in a cloud secret manager, including support
for:
o MIMB THIRD-PARTY SOFTWARE UPDATES:
· All third-party & open source software has been upgraded to their latest versions for bug fixes, improvements, and better security vulnerability protection. For more details, see the published MIMB's Third-Party & Open Source Software Usage and LICENSES.