Analytics

The Latest

15 March 2022: Snowflake announced its planned acquisition of data applications builder Streamlit. Snowflake’s goal is to integrate app building into its Warehouse-as-a-Service platform with simplified data access and governance features. 

Why it’s Important

There has been a growing trend in the acquisitions of analytics platform developers to boost product features and improve capabilities of data science tools.

IBRS expects more mergers and acquisitions among leading Cloud analytics vendors that will commence the initial stages of consolidating the hyperscale, elastic analytics market. It projects more integration of key components of the data analytics system in the next three years. In particular, data catalogues or data sharing solutions will become increasingly integrated with Cloud data lakes and data warehouses.

However, it is the use of centralised data repositories - data lakes and warehouses - to simplify the development of low-code apps that has been overlooked. One of the biggest challenges and costs for low-code apps development is integration. However, data analytics platforms have already integrated and normalised data from multiple systems. As a result, using these centralised data resources for low-code application development could be very attractive. 

Microsoft’s Dataverse is essentially this concept - albeit within the Microsoft world. Snowflake’s investments in Streamlit are an indication that there is a growing market for this use case.

Who’s impacted

• COO, CIO, CTO
• Business analysts

What’s Next?

Organisations should look at how their low-code initiatives tie into data analytics initiatives. Low-code platforms generate not only data captured from forms, but also metrics on how processes are performing - data which will likely end up being reported upon via analytics platforms. But there are also opportunities to leverage the analytics platforms to act as engines for low-code and rapid application development environments. Bringing the people involved in each of these areas together can reveal new opportunities to ‘streamline the process of streamlining processes’.

Related IBRS Advisory

1. VENDORiQ: AWS Accelerates Cloud Analytics with Custom Hardware

The Latest

15 March 2022: Google announced general availability of Dataplex, a ‘dash fabric’ (aka, data mesh) solution that allows enterprises to centrally manage and control data across data lakes, databases and data warehouses. Google claims that the product can mesh Google Cloud with open source technology for analytics professionals to better govern data. Dataplex was launched together with Datastream and Analytics Hub that make up Google Cloud’s database services in its analytics portfolio.

Why it’s Important

Since it was introduced in 2019 by its creator Zhamak Dehghani, the concept of data mesh is becoming hyped. Similar to service oriented architecture (SOA), it is misunderstood by vendors and buyers alike who believe that it is all about technology. Instead, data mesh is as much a philosophical shift, from viewing data being centralised in data lakes or warehouses, to managing data close to where it is created, through a domain-oriented design with a self-serve data infrastructure.

Google, on the other hand, identifies Dataplex as a data fabric, which provides a technology layer over disparate data sources for better access, discovery, integration, governance and security. Data fabric focuses on existing multiple centralised technologies that consolidate data. Data mesh, on the other hand, promises a fully domain-oriented, decentralised approach, since it considers all enterprise data as a set of different repositories, preventing any loss in domain expertise during translation, unlike with a data lake. Thus, in a pure data mesh platform, different groups can manage data as they see fit, sharing some common governance measures while maintaining their domain knowledge on the data.

IBRS has observed that data catalogue vendors are leveraging data mesh rhetoric to market their products. However, most of these do not truly align with the philosophy of data mesh, which is fine for the near term, as few organisations are prepared for the changes involved when adopting such a democratised approach to data management.

Who’s impacted

• CTO
• Analytics teams

What’s Next?

Organisations that want to explore data mesh concepts must carefully consider shifts in data team structures, roles, responsibilities and skills before looking at technical solutions.Some changes brought by such a data architecture approach will impact domain-specific variations in data across departments, domain ownership, data product self-containment, and governance architecture to preserve global controls.

Related IBRS Advisory
1. Business First Data Analytics - Webinar and Q&A

The Latest

22 February 2022: MetricStream has launched software solutions for governance, risk and compliance (GRC) that generate quantified, AI-powered risk insights for business growth, cybersecurity, and environmental, social and governance (ESG) reporting compliance. The SaaS company’s line of GRC software products address enterprises’ manual processes for GRC reporting with automation and improved visibility. The solutions consolidate fragmented and siloed data sources required to report on GRC.The solutions are available as three pre-configured packages, with the end goal being to enhance enterprise ESG scores.

Why it’s Important

Organisations with a lack of GRC capabilities can surfer from weaker strategic and operational processes. Without clear accountability and ownership, they run the risk of operating outside compliance boundaries, potentially with penalties and regulatory sanctions.

The purpose of GRC is to provide a centralised risk repository and reporting, in theory, leading to better transparency through enterprise regulation measures.

While it is possible to implement GRC within existing business intelligence and data management tools, not all Australian organisations can deploy GRC this way due to limited expertise and capacity constraints within the analytics teams. Furthermore, unlike in large enterprises where robust BI tools are integrated into their core information repositories and external data sources, small and medium enterprises have yet to achieve a more mature data management capability, and lack the budget for analytics and information management teams. In the end, compliance reporting costs them a lot of their financial resources to be at par with the quality of reporting that regulatory offices demand from them. Pre-configured GRC and ESG reporting tools may be a more viable option for these enterprises.

IBRS believes that GRC is becoming increasingly important among Australian organisations and will impact them across industries in terms of transparency through systemic workflows where real-time insights can be used to guide decision-making that meets minimum requirements from regulatory changes.

Who’s impacted

• COO, CIO, CTO
• Business analysts
• Risk managers

What’s Next?

Organisations need to be familiar with GRC and how they can best create a culture of compliance that ensures active oversight and adherence to applicable laws and regulations. Senior executives can drive a culture of transparency and efficient risk management by engaging in programs that meet GRC expectations, through compliance participation and implementation of preventive measures. This will improve risk control and promote good governance and organisational ethics.

To overcome the complexity of ‘build-it-yourself’ GRC and ESG reporting, consider if GRC software tools may complement the organisation's existing analytics platform through add-on solutions or dedicated products that make it easier to produce audit, accreditation and governance risk management reports.

Related IBRS Advisory

1. IBM Acquires Data Analytics Firm Envizi
2. More Evidence for Cloud Leading Sustainable ICT Charge

The Latest

25 January 2022: IBM has announced its acquisition of Sydney-based data analytics software company Envizi. In an official press release, the move was finalised to boost IBM’s capabilities to provide environmental, social and governance (ESG) analytics, which is an emerging specialised field.  

Envizi will be integrated with IBM’s existing package of manufacturing and supply chain solutions such as IBM Maximo, IBM Sterling, IBM Environmental Intelligence Suite (EIS) and IBM Turbonomic to support feedback automation in their operations and corporate environmental initiatives. 

Why it’s Important.

IBRS has observed increased activity by large vendors acquiring small, local Australian enterprises that specialise in data analytics. Some of these include the following:

• Fujitsu’s acquisition of Melbourne-based data and analytics firm Versor in 2021
• Cognizant’s 2021 purchase of Sydney-based Servian, a data analytics and AI vendor
• Healthcare tech firm Beamtree’s acquisition of New South Wales-based comparative analytics enterprise Potential(x) in 2021
• Accenture’s 2019 purchase of Australian big data and analytics consultancy Analytics8 then its series of acquisitions involving advanced analytics firms overseas such as Bridgei2i and Byte Prophecy in India, Novetta Solutions and End-to-End Analytics in the United States, as well as PRAGSIS BIDOOP in Spain.

Aside from these, acquisitions of data analytics startups by other firms outside of Australia have become prominent in the industry with the likes of Capgemini on Sweden-based Advectas, Genpact on Enquero, and Infogain on Absolutdata, which were all formalised in 2020.

IBRS believes that while it is beneficial for the industry to have vendors expand their analytics capabilities, customers or enterprise partners need to constantly assess the likely impact on their existing service contracts with analytics partner vendors. Some of the areas that are critical include terms and conditions, possible pricing changes, future services, contracted support and personnel changes, among others.

Who’s impacted

• CIO
• Development team leads
• Business analysts

What’s Next?

Organisations need to be prepared for their analytics partners to be the next targets for acquisitions. As part of its strategy, organisations must remain vigilant and engaged with their analytics vendor partners regarding any acquisitions and the potential impact on services and costs. This includes assessing the implications of the potential scenarios that are most likely to occur, as well as the risks or opportunities that may be present with regard to adjusting to ramifications to the existing service, if there are any. Some potential risks or challenges that must be reviewed by the organisation’s legal and procurement teams can be found on this checklist.

Finally, organisations need to be cautious on assurances that are critical to their operations if these have not yet been put into written agreement. Becoming more pragmatic about the new vendor will minimise service disruptions in the future.

Related IBRS Advisory

1. Mergers & acquisitions require federated service providers governance
2. Mergers and Acquisitions - Devising the Right Strategy for IT

The Latest

09 November 2021: Amazon Web Services (AWS) announced the availability of Babelfish for Amazon Aurora. Babelfish enables its hyperscale Aurora relational database service to understand Microsoft SQL Server and PostgreSQL commands. This allows customers to run applications written for Microsoft SQL Server directly on Amazon Aurora with minimal modifications in the code. 

Why it’s Important.

This new feature in Amazon Aurora, means enterprises with legacy applications can migrate to the Cloud without the time, effort and huge costs involved in rewriting application codes. In addition, using Babelfish benefits organisations through:

• Reduced migration costs and no expensive lock-in licensing terms, unlike in commercial-grade databases
• No interruption in existing Microsoft SQL Server database use since Babelfish can handle the TDS network protocol
• Availability of the open-source version of Babelfish for PostgreSQL on GitHub under the permissive Apache 2.0 and PostgreSQL licenses 

Who’s impacted

• CIO
• Development team leads
• Business analysts

What’s Next?

More general availability of hyperscale Cloud computing to support scalability and high-performance needs is expected in the coming months from major vendors. The most successful ones will require minimal changes in enterprises' existing SQL Server application code, speed of migration, and ease of switching to other tools post-migration.

Related IBRS Advisory

1. VENDORiQ: Google Next: Data - PostgreSQL Spanning the Globe
2. VENDORiQ: Google introduces Database Migration Service

The Latest

02 November 2021: Snowflake recently released the Snowflake Media Data Cloud that allows access to real-time, ready-to-query data products, and services from more than 175 data providers. The data-sharing company announced that its product can combine consumer data across sectors to reduce data latency and improve accuracy.

Why it’s Important.

More Australian organisations now recognise that access to external data enables enterprises to create one-to-one or one-to-many relationships for more reliable insights into data. Since it is difficult for businesses to make sense of data they don’t generate themselves, sharing information between internal business units inside the same company or between outside organisations, has narrowed insight gaps aside from lowering the cost of data collection and research. Some recent developments in this area include the following institutions that have extended their data sharing:

• In 2014, Coles revealed that its online shoppers using Flybuys would have their personal information shared with 30 companies under the same Coles umbrella as well as with third parties in more than 23 countries.
• Woolworths first started granting access to its consumer shopping behaviour data with all of its suppliers in 2017 to support collaborative decision-making with a customer-centric approach. However, it remains obstinate against disclosing all companies that handle its data when asked to submit comments during the Privacy Act review in 2021.
• In June 2021, Bunnings announced an upgrade of its tech platform to capture customer information to improve buyer experience. Its privacy policy page explicitly discusses how information is shared with third party businesses such as financial searches, security providers, market research firms, and payment collectors.
• Likewise, Target Australia discloses customer information to its service providers based overseas and to external call centres, recruitment companies and external fulfilment businesses. 

Ensuring the rights of consumers whose data is being shared can be an issue and apprehensions about maintaining privacy and confidentiality are often raised. The government introduced open banking across the country to provide consumers greater control of their personal data, and with whom it is shared, when applying for banking services.

Enterprises in the data-sharing environment must also find ways to ensure fair and equitable advantage of the information by accessing the same level of data insights as their competitors do. 

Who’s impacted

• CIO
• Development team leads
• Business analysts

What’s Next?

Enterprises need to address the challenges of sharing large scale datasets, such as adherence to legislative and ethical frameworks, using personally identifiable information (PII) for testing, defining the critical role of service providers and their limitations, and improving the overall context of each shared data environment. This can be achieved if policies, procedures and standards on data privacy and security are aligned with data ethics that engender trust among the myriad direct and indirect actors involved in data sharing. Whatever goals such practice entails (such as developing innovative ancillary products with business partners or improving customer care by analysing real-time dashboards for rapid issue resolution), making the best use of opportunities in the field needs to be secure, lawful, just and ethical to ensure that collaboration leads to better decision making when building upon the work of others and fostering a culture of trust. 

Related IBRS Advisory

1. Beyond privacy to trust: The need for enterprise data ethics
2. Three ways to turn employee engagement results into actionable and achievable plans
3. Data loss by the back door, slipping away unnoticed
4. How Australia must use the PageUp data breach to become stronger - AFR - 18th June 2018

Too often, information communications technology (ICT) and business analytics groups focus on business intelligence and analytics architectures and do not explore the organisational behaviours that are required to take full advantage of such solutions. There is a growing recognition that data literacy (a subset of digital workforce maturity) is just as important, if not more important, than the solutions being deployed. This is especially true for organisations embracing self-service analytics.

The trend is to give self-service analytics platforms to management that are making critical business decisions. However, this trend also requires managers to be trained in not just the tools and platforms, but in understanding how to ask meaningful questions, select appropriate data (avoiding bias and cherry-picking), and how to apply the principles of scientific thinking to analysis.

Download the pdf now.

Staff_Need_Data_Literacy_Presentation_Kit_-_IBRS.pdf

The Latest: 

26 June 2021: Zoho briefed IBRS on Zoho DataPrep, it’s new business-user focused data preparation which is being included in its existing Zoho Analytics tool, as well as being available separately as a tool to clean, transform and migrate data. DataPrep is in beta, and will be officially launched on 13th July 2021.

Why it’s Important

Traditionally, cleaning and transforming data for use in analytics platforms has involved scripting and complex ETL (extract, transform and load) processes. This was a barrier to allowing business stakeholders to take advantage of analytics. However, several analytics vendors (most notably Microsoft, Tableau, Qlik, Snowflake, Domo, etc.) have pioneered powerful, drag-and-drop low-code ETL into their products.  

Zoho, which is better known for its CRM, has an existing data analytics platform with Cloud storage, visualisation and reports, and dashboards. While the product is not as sophisticated as its top-drawer rivals, it can be considered ‘good enough’ for many business user’s needs. Most significantly, Zoho Analytics benefits from attractive licensing, including the ability to share reports and interactive dashboards both within an organisation and externally. 

However, Zoho Analytics lacked a business-user-friendly, low-code ELT environment, instead relying on SQL scripting. Zoho DataPrep fills this gap by providing a dedicated, AI-enabled platform for extracting data from a variety of sources, allowing data cleaning and transformations to be applied, with results being pushed into another database, data warehouse and Zoho Analytics. 

All existing Zoho Analytics clients will receive Zoho DataPrep with no change to licensing.

However, what is interesting here is Zoho’s decision to offer its DataPrep platform independent of its Analytics platform. This allows business stakeholders to use the platform as a tool to solve migration and data cleaning, not just analytics. 

IBRS’s initial tests of Zoho DataPrep suggest that it has some way to go before it can compete with the ready-made integration capabilities of Tableau, Power BI, Qlik, and others. In addition, it offers less complex ETL than it’s better established rivals. But, that may not be an issue for organisations where staff have limited data literacy maturity, or where analytics requirements are relatively straightforward.

Who’s impacted

• CIO
• Development team leads
• Business analysts

What’s Next?

The bigger take out from Zoho’s announcement is that ETL, along with all other aspects of business intelligence and analytics, will be both low-code, business-user friendly and reside in the Cloud. ICT departments seeking to create ‘best of breed’ business intelligence architectures that demand highly specialised skills will simply be bypassed, due to their lack of agility. While there will be a role for highly skilled statisticians, data scientists, and machine learning professionals, the days of needing ICT staff that specialise in specific reporting and data warehousing products is passing. 

Related IBRS Advisory

1. Snowflake Gets PROTECTED Status Security Tick by Aussie Auditor
2. IBRSiQ: Power BI vs Tableau
3. Business-First Data Analytics
4. AWS Accelerates Cloud Analytics with Custom Hardware
5. IBRSiQ AIS and Power BI Initiatives
6. Trends in Data Catalogues
7. When Does Power BI Deliver Power to the People?
8. Staff need data literacy – Here’s how to help them get it

The Latest

26 May 2021: Google has introduced Datasteam, which the vendor defines as a “change data capture and replication service”. In short, the service allows changes in one data source to be replicated to other data sources in near real time. The service currently connects with Oracle and MySQL databases and a slew of Google Cloud services, including BigQuery, Cloud SQL, Cloud Storage, Spanner, and so forth.

Uses for such a service include: updating a data lake or similar repository with data being added to a production database, keeping disparate databases of different types in sync, consolidating global organisation information back to a central repository.

Datastream is based on Cloud functions - or serverless - architecture. This is significant, as it allows for scale-independent integration.

Why it’s Important

Ingesting data scale into Cloud-based data lakes is a challenge and can be costly. Even simple ingestion where data requires little in the way of transformation can be costly when run through a full ETL service. By leveraging serverless functions, Datastream has the potential to significantly lower the cost and improve performance of bringing large volumes of rapidly changing data into a data lake (or an SQL database which is being used as a pseudo data lake). 

Using serverless to improve the performance and economics of large scale data ingestion is not a new approach. IBRS interviewed the architecture of a major global streaming service in 2017 regarding how they moved from an integration platform to leveraging AWS Kinesis data pipelines and hand-coded serverless functions, and to achieve more or less the same thing that Google Datastream is providing. 

As organisations migrate to Cloud analytics, the ability to rapidly replicate large data sets will grow. Serverless architecture will emerge as an important pattern.

Who’s impacted

• Analytics architecture leads
• Integration teams
• Enterprise architecture teams

What’s Next?

Become familiar with the potential to use serverless / cloud function as a ‘glue’ within your organisation’s Cloud architecture. 

Look for opportunities to leverage serverless when designing your organisations next analytics platform. 

Related IBRS Advisory

1. Serverless Programming: Should your software development teams be exploring it?
2. VENDORiQ: Google introduces Database Migration Service

Conclusion:

Too often, information communications technology (ICT) and business analytics groups focus on business intelligence and analytics architectures and do not explore the organisational behaviours that are required to take full advantage of such solutions. There is a growing recognition that data literacy (a subset of digital workforce maturity¹) is just as important, if not more important, than the solutions being deployed. This is especially true for organisations embracing self-service analytics².

The trend is to give self-service analytics platforms to management that are making critical business decisions. However, this trend also requires managers to be trained in not just the tools and platforms, but in understanding how to ask meaningful questions, select appropriate data (avoiding bias and cherry-picking), and how to apply the principles of scientific thinking to analysis.

Conclusion: Regardless of its digital strategy, many organisations have not been positioned to properly leverage the digital and data assets that are available to them. A Chief Data Officer (CDO) role can improve this situation by advancing an organisation’s data portfolio, curating and making appropriate data visible and actionable.

The CDO position is appropriate for all larger organisations, and small-to-large organisations focused on data-driven decision-making and innovation. These organisations benefit from a point person overseeing data management, data quality, and data strategy. CDOs are also responsible for developing a culture that supports data analytics and business intelligence, and the process of drawing valuable insights from data. In summary, they are responsible for improving data literacy within the organisation.

The Latest

19 Nov 2020: During its annual summit, Snowflake announces a series of new capabilities: a development environment called Snowpark, support for unstructured media, row-level security for improved data governance and a data market.

Why it’s Important

Of Snowflake’s recent announcements, Snowpark clearly reveals the vendor’s strategy to leverage its Cloud analytics platform to enable the development of data-intensive applications. Snowpark allows developers to write applications in their preferred languages to access information in the Snowflake data platform.

This represents an inversion of how business intelligence / analytics teams have traditionally viewed the role of a data warehouse. The rise of data warehouses was driven by limitations in computing performance: heavy analytical workloads were shifted to a dedicated platform so that application performance would not be impacted by limits of database, storage and compute power. With Cloud-native data platform architectures that remove these limitations, it is now possible to leverage the data warehouse (or at least, the analogue of what the data warehouse has become) to service applications.

Who’s Impacted

Development teams
Business intelligence / analytics architects

What’s Next?

Snowflake's strategy is evidence of a seismic shift in data analytics architecture. Along with Domo, AWS, Microsoft Azure, Google and other Cloud-based data platforms that take advantage of highly scalable, federated architectures, Snowflake is empowering a flip in how data can be leveraged. To take advantage of this flip, organisations should rethink the structure and roles within BI / analytics teams. IBRS has noted that many organisations continue to invest heavily in building their BI / analytics architecture with individual best-of-breed solutions (storage, databases, warehouse, analytics tools, etc), while placing less focus on the data scientists and business domain experts. With access to elastic Cloud platforms, organisations can reverse this focus - putting the business specialists and data scientists in the lead. 

Related IBRS Advisory
Workforce transformation: The four operating models of business intelligence
Key lessons from the executive roundtable on data, analytics and business value

Conclusion: Artificial intelligence technologies are available in various places such as robotic process automation (RPA), virtual agents and analytics. The purpose of this paper is to provide an AI maturity model in the analytics space. The proposed maturity model can be applied to any type of industry. It provides a roadmap to help improve business performance in the following areas:

• Running the business (RTB): Provide executives with sufficient information to make informed decisions about running the business and staying competitive.
• Growing the business (GTB): Provides information about growing the business in various geographies without changing the current services and products.
• Transforming the business (TTB): Provides information to develop and release new products and services ahead of competitors.