Learnings from the field….
Not all projects are created equal. Some projects are more complex than others. Some projects are more focused on the user, others are focused more on the organization. Some projects have very specific needs, and some projects are more about exploration. Depending on the outcomes of the project, your approach and style will change. This has never been truer than when it comes to data projects and software projects.
In this article, we will review the major factors when comparing Data Warehouse Projects and Software Projects. Understanding how these projects differ can help manage the expectations of the stakeholders and ensure the successful completion of these projects.
Data warehousing (DW) is a method that is used for compiling and examining data from many sources to gain insightful business knowledge. A data warehouse often combines and analyzes corporate data from numerous sources. The business intelligence (BI) system, which can analyze and report on data, is built on the foundation of the data warehouse. This makes it crucial for each new data warehouse project to be completed properly; otherwise, the rest of your BI infrastructure will suffer.
A software project is an entire process of developing software, from gathering requirements to testing and maintenance, carried out in accordance with execution techniques over a predetermined amount of time to produce the desired software output.
It is crucial to consider the larger picture while developing any new data warehouses. Think about how your design will fit into a future organization-wide data warehouse as well as how it will address the current business issue. By taking more effort up front, major design modifications to accommodate future capabilities can be avoided. If you look a little closer, you might discover that some of the present reports are not being utilized anymore or that people would prefer a different kind of solution even though they are now using the data in a particular way. It is harder to change a data warehouse when requirements change.
“Most of the time software projects are implemented to solve a specific problem or objective.”
There are no requirements to predict future use-cases in the initial development as that is not the purpose. As the requirements evolve, these additional requirements are added as features. It is not assumed that changing components or features of a software project is easier than that of a data warehouse project.
Projects involving data warehouses have unique needs in terms of the physical design of the database system. These specifications set the data warehouse projects apart from the operating data stores and are frequently overlooked. Software projects still work with data. But these databases are built to be optimized for the application that it is supporting. It is not optimized for analytics. Analysts would go into the data infrastructure and find it very difficult to locate the data and then cleanse the data for their analytics. Data warehouses are already in a format ideal for analysts with all the business rules and measures implemented. Creating views and then connecting them to BI solutions will ensure that all users work with the same curated data.
“Data warehouse projects follow specific patterns and rules and there are very few differences from organization to organization.”
You need to extract the data from the various sources, load it into a staging area, transform the data and load it into a data warehouse. From there you need to serve the data to the various users and business units. Your approach can differ in some places as well as the techniques but in the end, the purpose is to get usable data to the end-users.
The real work and the most complex part of the development is the modeling of the data. A thorough and optimal data model aids in the development of a streamlined, logical database that removes redundancy, lowers storage needs, and facilitates effective retrieval. Data warehouses are less about the technical implementation and more about getting the data in the right format to be usable. The technical implementation only aids in the performance of that data modeling and getting the data to the right users.
A software project is very use-case specific and can differ significantly based on the requirements of the customer. The approach, the technology, and the language are implemented to solve the functional requirements of the users. Software projects might follow some sort of development methodology like Scrum or Kanban, but the decisions made throughout the development are in support of the delivery of the solution.
The most important factor to note is the ownership of the project. In the past, be it a software project or a data warehouse project, that ownership resided with the IT department. What needs to be realized is the ownership of data warehouse projects should reside within the business.
Data warehouse projects have more to do with culture than technology.
Building a data-enabled organization is essential to accelerating the transition from the project-driven, ad hoc use of data to its fundamental operational integration. But making the switch from instinctual to data-driven decision-making isn't simple. Organizations must overcome a variety of obstacles, from culture to logistics, to use a data warehouse to their benefit. Data-driven decision-making is about leveraging the right talent to use the data most effectively. Establishing a data culture may be a never-ending task where the focus must be placed. That is when data warehouse projects start to fail.
Software projects are about tools to complete a task. Avoidance of the software will inhibit your ability to complete that task. It has less to do with culture and more to do with achieving your daily objectives. Yes, you would require some training and time to become comfortable with the solution and use it optimally, but it would not require a significant change in mindset.
There are similarities between data warehouse projects and software projects, but in the end, data warehouse projects are a means to an end, whereas software projects are about building things. Data warehouse development is more geared to analytical activities whereas software development has much more in common with traditional engineering. Understanding the difference between these two types of projects and what it requires to be successful, will add to the successful completion of both.
By Malcolm De Bruyn
Data Consultant
Kommentare