Data Migration - Problems We Solve

Problems that are usually encountered when dealing with Data Migration are far from being trivial. The existing statistics is impressive:

Here are some excerpts from the Standish Group and their well-known "Migrate Headaches" report:

According to The Standish Group, for a typical IT project involving the implementation of a data warehouse, 50% of the budget goes toward understanding the data, 20% to data movement, and 30% to application development. IT executives in the focus groups corroborated this, noting that 60-80% of the time spent on data migration consists of trying to understand the data, and only a fraction of the time is spent physically moving it.

in 1998, 74 percent of all IT projects either overran or failed, resulting in almost $100 billion in unexpected costs. Of the 15,000 data migration projects that are starting in 1999, as many as 88 percent will either overrun or fail.

So why is that not so smooth, why projects have been so under-estimated and bring that much pain? Here's the list of problems that are common:

Knowledge problems

• 1. Documentation and experts are unavailable
• 2. Data Migration experts with live experience are rare

Data source problems

• 3. Complex structure in source systems: i.e. multiple data sources, multiple data formats and media (excel files, legacy formats, etc), data overlap across source systems
• 4. Complex target data structure or API undisclosed of target system
• 5. Source data inconsistent, incomplete, not up-to-date or duplicated
• 6. Hard-to-parse data sources: pictures, text files, MS Visio-diagrams that need to be transformed into well-formatted data storages

Management and progress-tracking problems

• 7. Hard to document the process - what aspects must be documented, in what format?
• 8. Hard to keep track of lost or failed data? What the reporting must include?
• 9. KPIs of the process - what are they, how to track them? How one understands that Data Migration process has completed successfully?
• 10. Target system storage structure is still under design, how to plan data migration process for that?
• 11. How to plan the overall process - what are the risks, how much time this or that normally takes, how many people/resources are needed?

IT problems

• 12. There are many Data Migration (ETL) software but it is expensive or comes with professional services.
• 13. Do one need or need not to migrate data in parts? How to break down data?
• 14. What if synchronization needed, that is source systems continue working after migration and timely data synchronization is needed after data migration first run.
• 15. The data sources are huge, how to achieve performance?

Please see details on Our Methodology for more details on how we solve Data Migration challenges