Connect with us

Tech News

Big data is useful, scary, and more subjective than you know

Big data helps us understand our customers, but it also helps budding companies sell information about you (or TO you), and is more subjective than you may know – it takes a human touch to determine what info is important.

Published

on

big data

big data

Big data is here and unavoidable

For years, we’ve written about big data and showcased the progression of business intelligence available now to brands of every size, in fact, most businesses have a feel for this type of data – open a spreadsheet of your sales data and you already know it’s just a bunch of numbers unless they are analyzed and filtered. Today, I want to review what big data is, how it is currently being used, what this means for the future, and most importantly, how it can be cherry picked and why it can upset entire industries.

[ba-pullquote align=”right”]”Big data” is typically consisting of at least dozens of terabytes in a single data set.[/ba-pullquote]“Big data” is defined as large data sets which cannot be managed with simple, common software that captures and processes the data, and is typically consisting of at least dozens of terabytes in a single data set. The challenges of big data are really big. It is described by Gartner analyst, Doug Laney as being three-dimensional, i.e. increasing volume (amount of data), velocity (speed of data in/out), and variety (range of data types, sources).

Let’s talk about how BIG this really is

Let me illustrate. The University of Nebraska physics department has 1.6 petabytes of data – that’s 1.6 million gigabytes in one department at one school. Boeing jet engines can produce 10 terabytes of operational information for every 30 minutes they turn. As of 2012, the average smartphone user has 736 pieces of personal data collected every day, stored for one to five years by service providers.

[ba-pullquote align=”right”]By 2020, there will be 5,200 gigabytes of data for every human on Earth.[/ba-pullquote]IBM’s chief executive, Virginia Rometty said, “By one estimate there will be 5,200 gigabytes of data for every human on the planet by 2020. And powerful new computing systems can store and make sense of it nearly instantaneously.” It has also been predicted that in the coming years, over 200,000 big data specialists will be required to make sense of the barrage of data being collected.

Big data is already being used today in a big way

Big data is a big deal and it’s not just because there’s a lot of it. In fact, today alone, SumAll raised $4 million and DataSift raised a whopping $42 million to help businesses make sense of their data as it relates to social media.

[ba-pullquote align=”right”]Retailers are analyzing your facial expressions on camera to tell if you’re a happy shopper, and tracking your gender, age, and size as you walk in the door.[/ba-pullquote]Big data is already used in amazing ways by the retail industry by analyzing shopper height and size as they walk in the door to determine age, gender, and more, and even have cameras analyzing facial expressions while you’re shopping to gauge your experience. If that doesn’t impress you, there’s already a seasoned company that is tracking “visual mentions” online so if you share a picture of your Starbucks cup on Instagram, even if you don’t say Starbucks or use GPS, Starbucks can see that their logo, even if curved, was used online on a social network.

Predicting the future with big data

But it’s not just that data is having a tremendous impact on life today, it is still a young sector with many startups yet to pop up to solve the data conundrums. SiftScience fights fraud using machine-learning that learns from data to recognize patterns of fraudulent behavior based on past examples, and Hadoop helps companies analyze massive amounts of generating about user behavior and their own operations while Recorded Future uses algorithms that unlock predictive signals based on web chatter to determine a brand anticipate risks and capitalize opportunities.

[ba-pullquote align=”right”]Intel is working on technology using big data to allow you to see three cars ahead, behind, and beside you.[/ba-pullquote]There are already projects in the works that allows forecasters to predict weather up to 42 days in advance, potentially saving lives and billions of dollars a year.Intel is working on a big data project that allows cars to communicate so drivers will be able to see three cars in front of, behind, and to the left and right – simultaneously. Ford is developing vehicle-to-vehicle and vehicle-to-infrastructure systems to warn drivers of potentially hazardous traffic events, like cars going through red lights.

But big data has some really big problems

First, and least upsetting, is that there are big problems with demographics, leaving brands with a lot of data that doesn’t yet mean much. Why? Incomplete self reporting is a huge issue because brands are still focused on using social networking profile data to gather intelligence on their site users, fans, and the like, but when they rely on this data, people may not be completely truthful (they may say they are 32, but they’re 12, and so forth). Additionally, privacy does protect users to a certain extent, blocking intelligence gathering by brands. Lastly, data is still largely inconsistent and unconnected – you may have a Twitter account and Facebook account, but a third party doesn’t know that unless (a) you use the same username consistently or (b) you grant access to both accounts through that third party.

While other problems exist (like how will we ever store all of this data, disseminate it, and make sense of it, and does it all really matter?), the biggest one we see is the potential for cherry picking, because when you look at a data set, it still takes a human to actually determine what is important to garner from that data set.

[ba-pullquote align=”right”]Big data may mean more information, but it also means more false information.[/ba-pullquote]Industry expert Nassim Taleb opined in February, “With big data, researchers have brought cherry-picking to an industrial level. Modernity provides too many variables, but too little data per variable. So the spurious relationships grow much, much faster than real information. In other words: Big data may mean more information, but it also means more false information.”

Taleb addresses something that could lead one to think that big data is faulty and bad, but perhaps Taleb is really pointing out the human nature that is still required in some instances of analyzing big data – and most people would not typically question a researcher or their methods, leaving analysis in its youngest phase subjective.

Chris Treadaway, CEO and Founder of Polygraph Media which is famous for data-driven analytics said, “To analyze big data, you have to know when you have enough data, know that you’re looking at the right data, and know how and when to draw conclusions from the data using methods developed from statistics theory and data science. That’s the great irony of “big data” – it’s as much of an art as a science, which is why the best efforts are multidisciplinary.”

“Big data can find tremendous hidden relationships,” Treadaway continued, “but you have to make sure your bias isn’t to find conclusions that don’t exist. Bias can cause the situation Taleb describes, and will cause disinformation as he says. If you’re cautious, discerning, and careful, you can make the most of big data. But there are pitfalls for the careless.”

And the coup de gras

[ba-pullquote align=”right”]Your performance data, finances, company info and more are already being repackaged for public consumption and monetization.[/ba-pullquote]The coup de gras is that professionals are being threatened by new ways big data is being used, but they are not recognizing it as a big data issue.

Several industries are seeing data about them individually, their performance, their company, their finances, all analyzed and repackaged for public consumption or monetization.

Imagine a site launches tomorrow based on publicly available data and you’re a social media consultant. Let’s say that this new site looks at who has recommended you on LinkedIn, Yelp, Angie’s List and so on, and has determined that the people recommending you are clients of yours, based on the assumption that it is the only reason they’d recommend you or review you. The new site also analyzes words and pictures used in your online bios to determine characteristics about you.

Then, they take those reviews and characteristics and quantify you into a score, giving you more points if someone from Coca Cola reviewed you than if the local dentist reviewed you, implying that you’re a higher quality consultant if you’ve worked with a major brand like Coca Cola than if you worked with a local dentist (God forbid you specialize in social media for independent medical professionals).

Then, Google gets interested in this new site and they invest, and later, they want to use that data to populate your Google+ profile, so now you, the social media consultant, has a score next to their face to determine how good you are at your job.

What’s wrong with that?

[ba-pullquote align=”right”]You must understand that data requires a human to determine what is relevant, which doesn’t always allow for the full context of the data points.[/ba-pullquote]Data is subjective, even when raw – it takes humans to determine what data points in the sea of data are relevant, and it doesn’t always take into account the context surrounding that data. You, the social media consultant, could have taken a two year sabbatical to execute social media strategies pro bono for three tiny charities, four local restaurants, two African orphanages, and a spa, earning a reputation for your high quality of work and compassion that can’t possibly quantified by a computer.

This scenario is fake. For now. But with every human generating billions of data points every year, evaluations are just the first of many steps in what is to come with big data – the data is now generated, and it is a race to see what can be displayed about you and your business so that companies can sell to you or repackage your data and sell it to someone else. Even your brand will be using big data to gain insights into your customers so you can better serve them.

[ba-pullquote align=”right”]The race is on to see what can be displayed online about you and your business, which is being repackaged and resold.[/ba-pullquote]There are pros and cons to big data, but the reality is that it is unavoidable, even if you ignore it or misunderstand it. Consumers need to begin to recognize when they see big data, and understand that it may not be the true context of that data, as it is ripe with humans’ decisions regarding what is important about a data set. This is just the beginning.

Continue Reading
Advertisement
4 Comments

4 Comments

  1. Patrick Gallagher

    December 6, 2013 at 12:43 pm

    100% spot on conclusion: “…as it is ripe with humans’ decisions regarding what is important about a data set.”

    I like how Rory Sutherland (sometimes with Taleb or speaking about his work) kicks these ideas around. As soon as you pick certain data points and make them *the* metrics to follow the data becomes skewed and meaningless. You changed it just by looking at it so hard.

    Good stuff.

  2. Hank Miller

    December 9, 2013 at 7:35 am

    We are drowning in data and it can lead to paralysis by analysis.

    Watching videos and researching how to make shrimp scampi, set a broken wrist or install a hard drive does not mean that you can do it. Somewhere along the line a human with experience in the appropriate field has to provide guidance and identify the key points.

    Piles of data are just that – without someone with the abilty to effectively apply the appropriate parts to the specific question at hand there is nothing. Nothing but confusion

  3. Pingback: Top venture capitalist explains how tech startups can stand out when seeking funding - The American Genius

  4. Pingback: Big data is watching you - some will panic, others will rejoice - The American Genius

Leave a Reply

Your email address will not be published. Required fields are marked *

Tech News

Will COVID-19 break the internet?

(TECH NEWS) Internet usage is obviously up right now, but what can that do to the infrastructure? Tech companies say it’s the websites and local networks that are slow.

Published

on

internet world slow down

With more people staying at home, working from home and doing school from home, the internet is being taxed. You might have noticed your own service running slower or low-quality video streaming. Do we have to worry about the internet crashing? The quick answer is – “It depends.”

Yes, Americans are stressing the internet

The internet is actually pretty resilient when it comes to bandwidth. The network cables that connect people to the internet are built to handle spikes in use. When you stream video, it’s designed to adjust to your connection for the best quality. Even though Netflix, YouTube, and Amazon Prime are reducing the download speeds in the Europe market, there is no reason to suspect that the internet is going to shut down during this crisis.

That being said, Tech Crunch reports that download speeds in the United States are being affected in some markets. New York City, one of the epicenters of the COVID-19 virus had download speeds drop by about 24%. Austin saw a drop of 44%.

Rural markets are struggling. It’s hard to imagine that there are still some places in the United States that don’t have internet access. Other places may get internet, but the service can be unreliable on a good day. With the added stress of people staying home, service can be even spottier.

Traffic might be up on the internet, but the system was built to scale up. Think about how much more data is available today over two decades ago. And consider how many more users there are from even 10 years ago. More Americans are streaming movies and TV shows than ever before.

It’s local networks and websites that may see a problem

Facebook CEO Mark Zuckerberg said,
We’re trying to make sure that we can stay in front of this challenge. Right now, this isn’t a massive outbreak in every country around the world, but if it gets there, then we really need to make sure we’re on top of this from an infrastructure perspective and make sure that we can continue to provide the level of service that people need in a time like this.”

Google, Amazon and Facebook have been built for spikes in usage, but even Amazon’s website had a problem in 2018 on Prime Day when their servers couldn’t handle the number of shoppers. Big companies have the infrastructure in place to deal with the kinks of added traffic. There could be some issues that come up, but it’s unlikely to shut down things for too long.

It’s more likely that users will see issues in local websites that aren’t designed for the added traffic. Home networks will be stressed with multiple people trying to manage work and school at the same time. If you’re experiencing problems, check how many devices are trying to access the system within your own home. Go with SD streaming instead of HD.

The Internet was built to withstand a nuclear bomb

One BuzzFeed article believes that the likelihood of the internet breaking down is low. There may be challenges in some areas, especially as more providers lift data caps for its users. But most companies are aware of the problem and are trying to ramp up services to meet demands during this crisis.

Continue Reading

Tech News

Help solve COVID-19 with your home computer

(TECH NEWS) Your home computer can do more than just show you funny cat pictures or get you in trouble with family members, it can help solve COVID-19.

Published

on

Folding@Home

Did you know that while you are sitting at home doing your social distancing thing, be that working from home, video games, movies and TV, or making funny videos out of boredom, you can actually be actively helping solve the COVID-19 outbreak?

If you have a computer of almost any quality, or even a specific series of Android phones, you can become part of a huge network of computers that process data from scientists and medical researchers. If the internet is ever going to serve a good purpose, this is it!

Basically you just need to download a small app called Folding@Home, choose the COVID-19 projects and packets of data will be sent to your device and it will process that data in the background of whatever else you use the computer or phone for. It’s free, easy, and practically invisible to your everyday life.

“So what will my computer be processing in the first place?” you may ask, well I’m glad I made you ask. Think of viruses as a robotic manufacturing piece of equipment, you can see what comes up to it, and what leaves it, but what does it actually affect and how? You can’t see its inner mechanical workings or the program running it, this is the information from Folding@Home your computer will help scientists to understand so that they can craft a vaccine.

Now a lot of new technology is helping battle this pandemic, like 3D printed masks and valves, disinfecting robots, along with just keeping the people happy like guaranteeing continued services, and I am not discrediting any of that, they are very important to stop the spread. This on the other hand goes for the jugular as it were, to stop the virus on its own home turf, and costs you almost nothing. (Technically it costs you more power, but hey with a lot of power companies not expecting payment, and not charging late fees and whatnot, maybe they take the brunt of this payment)

If you want to be able to go back into the world you by now miss, this is an easy way to contribute to helping everyone. You can say “I helped solve the COVID-19 crisis!” If doing this makes you feel good or at least interests you, there are dozens of other projects similar to Folding@Home through Berkeley Open Infrastructure for Network Computing(BOINC) that you can contribute to such as tracking asteroids, simulating star formation, breaking down data about other viruses and medical needs, even the search for alien radio signals.

In a time when we all need help, take a step forward and BE the help.

Continue Reading

Tech News

Instagram helps pass social isolation with co-watching

(TECH NEWS) As social distancing become commonplace, Instagram responds with co-watching. The Newest way to look at and watch content with friends.

Published

on

Instagram co-watching

Deep into the second week of quarantine, third or fourth week for some of us, the isolation is starting to become quite real. Thanks to modern technology we can reach out to our friends and family without leaving the house, but it pales in comparison to the social lives many of us once enjoyed together. While you can certainly FaceTime or video call your friends, it’s still difficult to watch things together, mimicking the in-person experience. Many people have begun searching for apps that allow you to watch televisions shows and trending news together, so you can all see the same thing, at roughly the same time (thanks, lagging) and comment accordingly.

In a timely release, Instagram just launched a new feature called “Co-Watching.” This takes Instagram from a solo experience to a shared experience for up to six people. Co-Watching gives users the ability to video chat and browse through Instagram’s content together, thus making it more of a social gathering. The only downside to this feature, in my opinion, is that you cannot Co-Watch IGTV. Oftentimes, IG posts that are over the time limit are shifted to IGTV and you won’t be able to watch the full post with Co-Watching, but all other feeds and content on Instagram will be able through the new Co-Watching feature (except private posts, of course).

Ready to Co-Watch? Getting started is pretty easy, if you’re somewhat familiar with Instagram. To start, initiate a video call with whomever you want, up to six, in your Co-Watching party, by tapping on the arrow icon in the upper-right corner and select the video camera icon. You’ll see the video chat interface pop up and from there you’ll want to look to the lower right-hand corner for a “media” button, which looks like a mountain photo icon. Tap on that icon and you’ll see all the posts you’ve liked. Select a post or video from your favorites, or from Instagram’s recommended feed and whatever you tap will be shared to all your partygoers. If you’re watching a video, it will continue to loop until you or one of your friends select something new.

There are several other group chat/watching options currently available if Instagram isn’t your jam. Netflix can be used with the Party app. Netflix Party is available on Chrome browsers (on desktops or laptops) and allows you to synch your favorite videos with group chat. There is also the Squad app. It allows you to screen share anything on your phone with your friends. This works with texts, IG, Snapchat, Spotify, YouTube, Amazon, TikTok, and more. Start a group video chat with your selected friends, then broadcast your screen and start chatting. Squad is available in the App Store and Google Play.

While Instagram’s new feature is fun, the inability to share while watching IGTV makes it fall a bit flat for me. Have you tried, or will you try Instagram’s Co-Watching feature?

Continue Reading
Advertisement

Our Great Partners

The
American Genius
news neatly in your inbox

Subscribe to our mailing list for news sent straight to your email inbox.

Emerging Stories

Get The American Genius
neatly in your inbox

Subscribe to get business and tech updates, breaking stories, and more!