current position:Home>Business growth has encountered a bottleneck?You must check out these methods of using big data to drive business growth

Business growth has encountered a bottleneck?You must check out these methods of using big data to drive business growth

2022-11-24 22:49:28fanstuck

目录

前言

一、海量数据处理方案

1.1 大数据存储

1.1.1 传统数据存储

1.1.2 Cloud big data warehouse

1.2 数据BI快速展示

1.2.1大数据BI产业链结构分析

1.2.2 Huawei one-stop large dataBI方案



前言

With the rapid development of big data and cloud native blends together the interaction,The technology will come again a new wave of thunderbolt of challenge and change.Using the cloud native technology is easy、动态调度、The advantages of the on-demand scaling,How to accelerate the iterative update frequency of large data base software and system、Ensure that the overall environment of the eventual consistency,How to provide database with extreme performance、Elastic and infinite extension ability,And the resulting operational mode transition how to deal with,以上种种,Are we now need to take the lead in the breakthrough of subject.

当下来看,Is not only a technology problem of the above,Many companies are still using old big data processing scheme,But the increasingly accumulated data,需求增多.Traditional data centers both in performance、效率,Or in the investment income、安全性,Has far cannot satisfy the needs of emerging applications,Be badly in need of new data center business big data processing center to support.In addition to the traditional high reliability、高冗余、Green energy-saving outside,New big data center also need should have virtualization、模块化、弹性扩展、Automation and a series of features,To meet the demand of application has the characteristics of big data.These unprecedented demand,Let storage system architecture and function of unprecedented changes have taken place in the.

对于企业来说,Big data challenges include:

  • Business department has no clear big data requirements
  • Internal data islands serious
  • Data availability is low、质量差
  • Data management technology and architecture related questions
  • 数据安全问题
  • Big data shortage
  • Open data and privacy weigh

The release of more rapid data value,A set of perfect solutions and systems is the key.And this is a leader in the field of huawei cloud one-stop big data solutions,Popular industry recognized.In this paper, from the traditional big data solutions and according to the characteristics of big data,To analyze why choose huawei cloud big data solutions.


一、海量数据处理方案

1.1 大数据存储

1.1.1 传统数据存储

Business systems often produce a large number of log,The traditional way is to use centralized storage.Centralized storage is refers to a mainframe or multiple hosts of center node,Data set is stored in the center node,并且整个系统的所有业务单元都集中部署在这个中心节点上, Data computing almost completely Rely on a taichung、大型的中心计算机的处理能力.系统所有的功能均由其集中处理.在集中式系统中,Each terminal or client only responsible for data input and output,And the data storage and control processing completely to the host to complete.The most obvious feature of the centralized storage is deployment structure is simple.On the whole environment now,Need to face the test:

  • 系统安全性:需要对抗DDoS攻击,传统自建数据库需要自行实现,且实现成本较高,同时也需自行修复数据库安全漏洞.
  • 硬件成本:为了能部署数据库,开发者需要购买费用高昂的数据库服务器.一次性的硬件成本很高.而且考虑到硬件损耗,后期还会有维修和更换硬件的成本.
  • 运维成本:需要花费大量的人力物力来维护硬件、维护操作系统、维护数据库软件.
  • 可扩展性:传统自建数据库采用物理硬盘作为存储空间,受单盘容量的限制,并不支持弹性升级,要增减节点,也比较难实现,如果某个数据库超过了这个存储的限制,Need operations and development of personnel data archive together,To reduce the database capacity.
  • 可靠性:单机实例,没有数据灾备.需要额外成本建立异地的数据灾备.不管是主从架构、还是负载均衡还是自动容灾方面,传统自建数据库都需要自行实现.

Traditional database or data warehouse products save any、查不出、扩容难、成本高的痛点.因此,How to the traditional database upgrade transition,Is one of the key issues facing the enterprises.

一方面是传统数据库不能满足互联网企业新需求.互联网企业因业务快速扩展,需实时处理海量数据,并发访问量非常大.传统数据库基于专用服务器和高端存储构成共享存储环境,设备造价昂贵,且性能和扩展性极其有限,无法满足互联网行业对数据库高并发、实时在线需求.

另一方面是传统数据库不能适应云计算技术新需求.云计算公司依托技术积累,发起由IOE架构向基于廉价PC服务器技术的变革,降低IT服务支出,倡导以软件为中心的架构演变,降低数据库维护升级的风险.传统数据库环境是集中在一个地点的高稳定、高可靠、High availability of the high-end enterprise equipment,制约云计算企业未来业务发展,增加长期运营成本.

Traditional data storage is facing the following four questions:

  • 无法应对每秒上万次的读写请求,硬盘IO此时也将变为性能瓶颈.
  • 表中存储记录数量有限,横向可扩展能力有限,纵向数据可承受能力也是有限的,面对海量数据,势必涉及到分库分表,难以维护.大数据查询SQL效率极低,数据量到达一定程度时,查询时间会呈指数级别增长.
  • 难以横向扩展,无法简单地通过增加硬件、服务节点来提高系统性能.
  • 对于需要24小时不间断提供服务的网站来说,数据库升级、扩展将是一件十分麻烦的事,往往需要停机维护、数据迁移.为了避免服务间断,如果网站使用服务器集群,则根据集群策略,需要相应的考虑主从一致性、集群扩展性等一系列问题.

So combining the existing cloud native and big data technology,The data warehouse, and what form it is again?

1.1.2 Cloud big data warehouse

In huawei cloud data warehouse GaussDB(DWS)A new generation of cloud several positions, for example,GaussDB(DWS)As a new generation of fully scene data warehouse,具备极致性能、高扩展、极简易用、One-stop analysis features such as,Meet the number of enterprises to build the new era of big data warehouse requirements.融合分析能力是云原生数据仓库GaussDB(DWS)核心亮点之一.GaussDB(DWS)采用一套SQL引擎,支持Oracle、Mysql、HDFS等多源数据融合分析,并通过算子下推、加速集群等技术对分析性能进行了大幅优化,在数据免搬迁的前提下,实现了跨源数据免搬迁、高效分析.

 

Can carry on the on-demand at runtime extension ability is the prerequisite for any business to grow.因为这种能力让企业可以专注于追求商业目标,而不用担心存储空间大小的限制.传统数据库将所有文件和资源都存储在同一主机中,而云原生数据库GaussDB则不同,它不仅允许你以不同的方式存储,And is not affected by storage space problem.

比如我们从MySQLThe database data migration toGaussDB:

1.导出mysql数据

 show variables like '%secure%' #查询出secure_file_priv地址;

2.下载数据并上传至GaussDB服务器

3.导入GaussDB

COPY I18N_MESSAGE FROM '/home/gaussdba/app/bin/temp.dat' WITH DELIMITER '''';

 

十分方便便捷,GaussDB 提供配套的迁移工具,支持使用dump和loadExport data from the database into standardized general file,可使用GaussDB的 load导入Oracle、MySQL、SQLserverSuch as database export file,同时也可以使用dump导出GaussDB的数据,And then imported into the other database or otherGaussDB数据库,可支持Teradata、Oracle/Exadata、GreenplumData analysis system such as the smooth transfer of,而且支持SQL 2003标准,兼容Oracle/TeradataPart of the grammar and data structure,支持全局事务、存储过程、数据强一致性保证.且GaussDB库仓一体,逻辑集群,Support a cluster to accommodate the data mart、数据仓库.

GaussDB(DWS)云原生数据仓库支持冷热数据多温存储,热数据存储于数仓内部,以获得良好的查询分析性能,冷数据可分级存储到更低成本的OBS中,不仅降低存储成本,并且在OBS内,通过合法鉴权,数据能够共享开放,供其他引擎处理分析,GaussDB(DWS)当前已经支持表内不同分区间的冷热数据存储,未来还将支持更细粒度、更加智能的冷热数据管理.

 Enterprise accumulation of huge amounts of data and various kinds of data assets,体量庞大,Need high performance large amount of data platform support from the whole data analysis and mining.依托DWS+BITools to build a global、直观的、关联性的、Visualization of operating the digital analysis platform ,Data analysis to drive business value improvement and upgrading management.

ba345733f5124839641e1a904672da1f.png

 

First is the requirement of high performance:Customer online business for insurance core business,The business need database with high concurrency、The ability to big table query,Especially the docking channels of the Internet and business,The database performance requirements is very high.In terms of policy bulk order scene,GaussDB(for MySQL)Because of its excellent performance and perfect support business core trading scenario.

而且GaussDB跨AZ部署高可用,数据安全可靠.This feature is at home,Friends and contrast agent is the leading,It can make it across the3AZ部署,Any node failure is not fatal influence on the business,Failover speed to do10秒以内,Do zero data loss,采用白名单、VPC网络、数据多副本存储等全方位的手段,对数据库数据访问、存储、管理等各个环节提供安全保障.且GaussDBQuick backup recovery,With the method of snapshot,相比传统MySQL物理备份,The recovery time several times.基于底层存储系统的多时间点特性,不需增量日志回放,可直接实现按时间点回滚.备份及恢复逻辑下沉到各存储节点,本地访问数据并直接与第三方存储系统交互,高并发高性能.

7a54043e331c8a7edfb58236a6434f47.png

 For real-time search、复杂BI SQLCustom parallel query function,After open as many tables associated query speed up10+ 倍.

1.2 数据BI快速展示

Now that talked about the dataBIThe business line then just tell him it.

1.2.1大数据BI产业链结构分析

大数据BIIndustrial chain structure on the whole, by the data source、数据获取、数据处理、Data analysis and data display five aspects constitute.

第一、From the perspective of data source

The source of data for large data application,Not only including the unstructured data,There are all kinds of system data,数据库数据.The unstructured data is mainly concentrated in the data on the Internet and some social networking sites and some equipment data,All of these constitute the source of data for large data application.For big data analysis tool,At this stage is for unstructured data analysis more.

BISystem is in the field of data integration technology is more and more mature,For the extraction of data,The request of a variety of data mining,Data integration platform can help enterprises to realize the data flow and interactive use,In the enterprise internal implementBIApplication is in order to can better the data share and use.

The data source made up of enterprise internal data and external data,企业的内部数据通常与具体业务紧密相关,And mostly we can control software system,如CRM、ERP或者HR系统.ETLResponsible for extracting data from the data source to the temporary middle layer after cleaning、转换、集成,最后加载到数据仓库中,成为联机分析处理、数据挖掘的基础.At last, through data display tools for data visualization analysis,满足各种应用需求.Most companies to big dataBIPlatform for data access control、Exploratory analysis independently and quickly build business functions such as data model with strong demand,The proportion of these function is selected as much as90%以上;

第二、From the perspective of development direction

BIThe development of to transform from the traditional business intelligence model,对于企业来说,BI不仅仅是一个IT项目,But also a management and way of thinking,From the technology deployment to the business process planning,BI迎来新的发展.对于大数据来说,At present more and more great attention in unstructured data,The emergence of different data analysis tool and the application range of the industry constantly increase,对于大数据应用来说,How the combination of industry and application of a deep is the most important thing.

第三、从工具的角度

传统BI使用的是ETL、数据仓库、OLAP、Visual report technology,Belong to the application layer and display technology,Are currently on the brink of elimination,Because it doesn't solve a lot of data(Including structured and unstructured)的处理问题.And big data applications is a complete technical system,包括用Hadoop、Stream processing technology to solve mass structured、Unstructured dataETL问题,用Hadoop、MPPTechniques such as mass data of calculation problem,用redis、HBASEAnd solve the problem of efficient reading,用ImpalaTechnologies such as on-line analysis, etc.

1.2.2 Huawei one-stop large dataBI方案

Based on huawei cloud data warehouse serviceGaussDB(DWS)及基础服务,Huawei one-stop large dataBI可以5Minutes to complete automatic reporting,Can easily get through multiple business system data,Integration of multi-source data,Build platform enterprise visualization analysis and decision,通过类ExcelThe designer of a key connection data source,拖拽字段绑定单元格,Statements can complete production.学习成本低,Products on the load,操作简便、快速,总结起来就是“Easy to learn and easy to use easy integration”.

方案优势

1、一站式

一站式大数据分析平台,包括数据采集、清洗、整合、存储、计算、建模、训练、展现、协作等,一个平台上完成全流程数据分析任务.

2、高性能

利用列存储和内存计算,Huge amounts of data second level response.AIIn-depth analysis of integration of a variety of algorithm,Mining potential value relationship、模式和趋势,构建数据模型,To make business forecast and insight.

3、极致易用

Launch cycle in weeks,The cloud platform support allOS,支持公有云、Private and hybrid cloud architecture.操作简单,业务人员也能轻松、快速地制作并分析数据报告.

4、移动跨屏

Seamless support all kinds of computer and mobile terminal system,并在这些终端设备上保持一致、易用的用户体验,Anytime and anywhere via mobile devices for data analysis.

 

The ultimate goal of digital transformation is not get the data,But the value of mining data,实现数据驱动业务增长.永洪BI定位于“面向业务人员的可视化分析”工具,Let zero based business personnel can independently complete the data analysis、Mining and insight,Make the data into the business,推动业务增长,实现降本增效.

 

华为DLVIs a one-stop data visualization development platform,适配云上云下多种数据源,提供丰富多样的2D、3D可视化组件,采用拖拽式自由布局,Designed to help fast customization and application of their own data screen.

1.丰富多样的可视化组件                                                                                                                            

提供丰富的可视化组件,Including the commonly used data chart、图形、控件等

2.专业级地理信息可视化                                                                

支持绘制地理轨迹、地理飞线、热力分布、地域区块、3D地球等效果,支持地理数据多层叠加

3.可视化编辑器

Drag and drop components can be completed through free configuration and the layout、所见即所得,No programming can easily set up visual screen,And according to the resolution on the equipment,Freedom to customize screen size

4.多种数据源支持

Seamless integration of huawei cloud data warehouse service、数据湖探索、关系型数据库、对象存储服务等,支持本地CSV、在线APIAnd the enterprise internal private cloud data

 

通过以上案例,我们不难发现,华为云大数据BIThrough the multi-dimensional data statistics way,Can better insight to the consumer's consumption behavior patterns and trends.As a new technology application scenario,大数据BIIs becoming one of the important direction of future business wisdom,Also will bring revolutionary change to traditional business.

With the advent of the era of new economy,越来越多的人开始意识到“数字经济”Has the great potential of,And big data is one of the key ring,Huawei cloud with resources and technology more strong advantage,Better able to provide customers with customized service differentiation,提升企业市场竞争力,获取更大市场份额.

通过以上案例,我们不难发现,华为云大数据BIThrough the multi-dimensional data statistics way,Can better insight to the consumer's consumption behavior patterns and trends.As a new technology application scenario,大数据BIIs becoming one of the important direction of future business wisdom,Also will bring revolutionary change to traditional business.

With the advent of the era of new economy,越来越多的人开始意识到“数字经济”Has the great potential of,And big data is one of the key ring,Huawei cloud with resources and technology more strong advantage,Better able to provide customers with customized service differentiation,提升企业市场竞争力,获取更大市场份额.

 

Just double huawei cloud11活动还在继续,截止到11月30日,访问活动网址就可领取10000Yuan full cut red envelope:

 还有数据库、服务器、YunAn congruent products also has a very powerful preferential,Hurry up of operate!                      

 

copyright notice
author[fanstuck],Please bring the original link to reprint, thank you.
https://en.cdmana.com/2022/328/202211242242345227.html

Random recommended