Starting and Growing a BI Department from Scratch

Sometimes we fall into opportunities that allow us to start fresh. Not much could be better in that regard than beginning and growing a…

Sometimes we fall into opportunities that allow us to start fresh. Not much could be better in that regard than beginning and growing a new business intelligence function in your company. I’ve learned a lot of lessons over the years. Here’s my advice.

Make sure your company can settle on one front-end tool. This is crucial. You should start simple — something that everyone can use with minimal training and can read data from the most popular databases. I recommend Pentaho as an open source contender.

Hire two people and make sure they are cross-trained. One person is your front-end specialist who can also do requirements documentation. The other person is your business analyst who can also program the front end. Make sure they are both lovable people who over communicate. Make sure everybody knows them and trusts them. You focus on the backend (assuming that you are the most technically proficient). That means you do DBA work + database design work. You build all of the ETL and you learn every data source in your company.

Your business analyst will start brainstorming what the various departments want in order to provide direction before they become requirements. The business systems analyst should know the roadmap to building what comes next. This person is responsible not only for requirements, but training and testing. Their reputation depends on delivering exactly what the customer asked for. Insure that every application has a release schedule and make hard calls about what’s in next and what’s pushed back.

My experience tells me that very few departments, even those hungry for data will hire full time systems analysts who will keep up with tech and implementation methodology. It might be controversial, but I say the systems analysts should stay with the tech team.

Your front-end specialist should know every trick in the book when it comes to changing fonts, colors, charts, pull-downs, filters, sql, writeback and securing reports for different groups. Give the people what they want. Make sure that when they navigate the system that they never get lost. Give the customers two or three ways to see the same thing, but ultimately narrow them down to one view.

You / your backend specialist should get the most powerful servers, the most capacious backup, the most comprehensive language (well, bash + Python and/or Go) to run your data management. Refine data designs and create alerts and alarms for when data that should be there doesn’t get there. You should know that your sources are failing before your customers (or at least within the first five minutes).

These are the three primary skillsets for a good BI department. You should be able to master them in 2 years starting from scratch.

The toughest question you will ever answer is “What were we thinking?” two years after an application was built. If you can answer that easily and get the same answer from both your department and the customers, you’re doing it right.