Why Data Science Sucks

Why Data Science SucksHottest Career of 2019! How Data Science Changed My Life Around.

Top 5 Strategies to Succeed in Data Science.

Adhaar SharmaBlockedUnblockFollowFollowingApr 4It seems like these days we are bombarded with articles on Data Science and how great and transformative it is supposed to be for your career or business.

I think it’s time to show some truths to better set expectations.

Because as my favorite blogger Tim Urban points out:Happiness = Reality — ExpectationsDISCLAIMER: I love data tremendously but I have gotten tired of people not knowing what they are getting themselves into with data science.

I also run a Data Quality startup aiming to reduce some of the suckiness.

Starting Off A ProjectWhat you think you will be doingPlanning and designing awesome algorithms and models to predict cool things because the data is perfect and ready to use.

Maybe you will use some of those fancy Neural Nets you learned in your recent Data Camp course.

Everyone knows writing on a whiteboard is mandatory if you are doing something smartWhat you will actually be doingFiguring out where the heck the data actually lives.

Realizing it is stored in a CRM that the marketing team owns and a database that the IT team owns.

Emailing back and forth with Carl from Marketing and Cheryl from IT to figure out why there are duplicate sets of customers.

And which is the source of truth.

Gathering all the different data sets.

Exploring the data and checking its quality.

Realizing you don’t know half the fields so emailing Carl and Cheryl some more.

And then cleaning, cleaning, cleaning.

The janitorial step of Data Science takes the most time.

Estimated to be 60–80%.

Building Your ModelWhat you think you will be doingImplementing your fancy Machine Learning algorithms in the latest awesome framework.

Maybe you will finally get to try out PyTorch.

Or build your own as you have always wanted to do.

What you will actually be doingSince you spent so much time dealing with data gathering and cleaning, the project deadline has crept closer.

So you revert back to the programming language, library and tools that you have used forever.

Then you find out the fancy Machine Learning algorithms aren’t giving you the results that you wanted.

You are on the verge of tearing your hair out so you try something simple like Linear Regression.

And voila! It works.

You start to question all the time you spent learning everything else trendy and new.

Using Your Model in the Real WorldWhat you think you will be doingOkay, you have finally developed a model that’s very accurate and is set to save the business a ton of time and money.

You hand it off to your engineering team, they deploy it right away and you are hailed as the hero for cutting costs in your department by 50%!This is how everyone celebrates after a project.

 Right?What you will actually be doingGoing around to all the necessary stakeholders to get their approval.

Bill from Finance is giving you a hard time because he thinks his team’s process might be affected.

You give Bill and his team 3 presentations that quell his fear.

You attend more meetings than your introverted self has the ability to handleThen the engineering team tells you they are short staffed and have to delay putting your model in production.

You are tired but believe in your work so you go to upper management and plead your case.

They finally agree.

Your model gets deployed and it does cut costs by 47.

83%.

You feel very proud and happy.

Until your product launches in Europe and you realize your model doesn’t generalize to European customers.

Back to the drawing board you go.

On a Serious NoteI hope this illustrates why Data Science sucks.

A lot of the activities really feel soul crushing and at times mind numbing.

Not to mention the imposter syndrome, the constant learning and the FOMO for the next big thing.

But those same activities could equally be considered fun depending on your personality, dreams, and goals.

I personally get joy after I am done Data Cleaning when everything is beautiful and perfect.

And I love that there is an endless stream of learning so I never get bored.

In the end, realize that there will be a lot of parts that will suck but if you can find satisfaction and pride in the process, as I do, it could be a career and track worth exploring.

Also, maybe build or start something that can make some of these things less of a pain to do :).

. More details

Leave a Reply