Big Data

Big Data and the challenge of managing information.

By Bruno Rossini, Cloud Director at Financial Solutions

Nowadays, the increasing amount of data forces us to focus on the importance of its interpretation and visualization; that is why we want to share an interesting video of David McCandless The beauty of data visualization” in Ted Talk, which although it is not a recent topic is something that does not lose validity.

In this Ted Talk, David McCandless talks about the importance of visualizing and correlating data, so that we can focus on the information that really matters to us.

The vast amount of data we handle today is ever-growing and data alone lacks any value and meaning if it is not in context, or worse, it could give us misinformation if we don’t correlate it properly.

One of the key points mentioned in this brief presentation is that by representing information graphically, we can detect or reveal patterns that help us to establish these correlations between different data sets and gain a broader perspective on the information we have that we might otherwise overlook.

It used to be said that data was the new fuel or “Data is the new oil”, but David McCandless reformulates the phrase “Data is the new soil”, this new interpretation tells us that data and information are extremely “fertile”, and it is possible to make them “flourish” all this with the help of a proper analysis through different tools and using our networks and connectivity.

The “information design” contributes a very important factor in the way to solve the problems of data interpretation and the interesting thing is that this design can help us to illuminate and enrich many aspects of daily life, either in real-time, such as traffic status, financial information, it can also be applicable in the short term, as in medical or meteorological factors, as well as in predictive situations such as commercial, criminal or epidemiological trends.

We know that the term “Big Data” refers to the fact that data has a volume and growth so fast that storage, management, and interpretation become a real challenge; and additionally, we cannot deny that also in terms of “Big Data”, the whole is greater than the sum of the parts, and finding a way to take advantage of it, will undoubtedly represent a very big competitive advantage.

According to research by New Vantage Partners (“Big Data Executive Survey – 2017”), while 85 percent of companies want to have a “data-driven” organizational culture and strategies, only 37 percent have successfully achieved this.

At Financial Solutions we have taken this challenge as an area of great value to help our clients not only to collect and store data but to process it based on different rules and correlations, presenting it in a graphical way to be more useful in decision making.

To achieve this we rely on a set of tools and services available from AWS, such as Kinesis, SNS, and SQS for real-time data collection, as well as SFTP, Cloudtrail, and Cloudwatch services that are part of the data ingestion sending their logs to S3 for further processing.

Although within AWS we also have multiple services that help us in the treatment and analysis of information, there are cases in which the transformation and processing of data are in charge of Logstash, which integrates directly with our elasticsearch stack.

With Elastic Cloud we manage the different indexes that will later feed the Kibana dashboards, making sense and correlating all the information that will be visualized in an attractive and friendly interface, based on a wide knowledge of UI/UX techniques, offering a great experience to the end-user.

Here is the video:

If you would like to learn more about how we can help you meet this challenge, leave us your contact information to schedule a call with one of our experts.

By Bruno Rossini, Infrastructure Director at Financial Solutions.

Related Post