How Have Recent Advancements in Big Data Been Impacting Businesses?

Building on this simple and effective connectivity, we have the rise of automation. The bot's making use of Big Data intelligence to service customer requests or to spot trends and make suggestions are commonplace now in retail platforms or videos services.

IoT Devices & the internet

Internet of Things (IoT) devices fuel the good stuff, the data!

IoT devices have provided vastly varied data sets (Surely I should have mentioned the 4 V's at some point!). Temperature, CO2, heat, heart (ECG), light, sound, and any number of sensors are cropping up all over. Every smartphone, every video camera is an IoT device producing vast volumes of data to be processed and interpreted.

All this data is so readily collectible now thanks to the rapid spread of network connectivity and the internet.

Developed economies continue to invest heavily in high-quality networks. In Australia, the NBN (National Broadband Network) has brought faster, more reliable connectivity to new homes and businesses. While less developed countries are leapfrogging the developed world by launching mobile (5G) networks capable of ultra-fast bandwidth speeds. They are avoiding maintenance and reliance on legacy equipment and Telecomms platforms by going to straight to mobile solutions.

So broad is the spread of the internet it's getting hard to find places outside of network range, areas to switch off, and be human again - but that is perhaps another article altogether.

The Impact

With Big Data comes big responsibility and big questions regarding security and more fundamentally, ethics.

From a security perspective, handling data, managing encryption at rest, in transit, while enabling people in your organization the freedom and autonomy to use the data can cause weighty burdens on security teams and Chief Information Security Officers (CISOs). Governments and regulators are often struggling to keep up; the reaction, Privacy laws design to prevent the evil actions and misuse of PII (Personally Identifiable Information) developed without the consideration for the consequence of implementation.

Even after we meet the security requirements, we must understand the biases wired into training data, potentially tainted with a lack of diversity and polluted with discrimination. We need to pose ethical questions; what should I understand about a person if I have harvested location data from their phone? Their heart rate data from their smartwatch? With whom they have met or connected with via social media. What have they purchased recently? Differential pricing or suggestions based on this type of data, or a combination, delivers a fine line between ethical and legal dilemma.

Closing statement

Literacy, IoT, and APIs have all generated phenomenal advances in the adoption of Big Data solutions, mostly for the increased bottom line, and some the betterment of society.

It is no longer the technological constraints that are slowing or preventing Big Data solutions seeing the light of day. It is arguably the security and ethics conversations surrounding big data that is having the most significant impact on businesses today.

My advice, consider the impact Big Data and technology has on the people in your team, organization, or personal life. Have the conversation and set some boundaries, draw your own ethical and moral lines before it is too late.