Support OzOriginal

Home Cases XML export
XML Export case
Print E-mail

Guerrilla IT Architecture

XML export case

1. Background

Company A is a provider of financial services and data processing. Company A stores data for one of its main applications in a non SQL database, for fast access and processing. One of Company A’s clients has requested data to be exported into an MS SQL Server database so they can apply their in-house analysis using their IT staff.

Two developers from two development teams were employed to implement the solution: one from a non SQL database team, which was the owner of the data and one Java developer.

2. Specifics and possible complexities

  1. The non SQL database might have a variable number of columns per table, so the structure might change on a monthly basis. In this case the exported database might have to be rebuilt due to schema changes.
  2. The data volume is in tens of gigabytes, which might create a problem to re-export/import when every small change occurs.
  3. Data has to be exported on weekly or monthly basis.

3. Initial solution

The solution came from the Java programmers team as following:

  1. The non SQL database programmers team was requested to create an XML export in a specific XML format, which incorporated columns structure close to SQL database.
  2. A standalone Java application was developed to import data into the MS SQL database from the XML generated by the other team.
  3. SAX was used for XML processing, a semi manual, low level way with stream pushback buffering.
  4. The import application is to be run client side, by client employees with XML file uploaded to client’s site.
  5. Additional complex logic for a “partial update” was developed.
  6. Application updates have to be installed on client site.

4. Results

After about half a year of development the application came to the following:

  1. 5 days for full import from XML file into MS SQL database.
  2. Additional “Partial update” functionality being very complex took much longer than expected to be developed and caused multiple issues. Eventually it was cut down only for simple cases with “full load” being used for most cases of data structure change. As it takes 4-5 days for each import, the usefulness for the client was questionable.
  3. Client site application updates and log analysis are hard to manage by local team.

5. A Note

This is a classic example of poor architecture, and the underappreciation of suggestion of proper architecture for replacement. Originally manager’s  expectations were to fix the biggest issues only. However once an architectural solution was presented, it was said it is “too obvious” and had to be done this way from the beginning and it cannot be considered as something “revolutionary”. This all occurred because the developer did not think long and hard about the solution from the beginning. However, no one could explain why for longer than a year nobody challenged the solution.

6. Architectural review

Requirement/Exiting Issue Architecture focus
Client site application use (run, upgrades, log analysis) is undesirable. Client pays for data being fully prepared by the provider and doesn’t have to find resources to run external application Focuses on in-house run approach

Too much of error prone complex code requires high maintenance

Focuses on out of the box solutions and as “less code’ to write approach

Specifics of not classic use of SAX increases maintenance headaches and learning curve for other programmers, makes more error prone

More classic programming, or no programming at all

7. The guerrilla architecture action

Solution

  1. Non SQL team to export data in CSV format ready for MS SQL Integration services. The change took 1 day by one developer for that team.
  2. Non SQL team to generate bulk insert, format file, create table, indexes, foreign keys based on the schema they have. Took 2 days.
  3. Create MS SQL integration services package for importing data. Use it to import data into MS SQL Server by client at client site. Company A provides data and MS SQL Integration services package to customers.

Notes

  1. Learning and proof of concept: 1 day by two developers in pair programming session. The following staging environment setup takes about an hour.
  2. Changing export to CSV sped up data generation by non SQL team.
  3. Half a year of Java development months of bug fixes ditched.
  4. During existing program analysis one of bugs was found that no disabling indexes and constraints while inserting gigabytes of data, which slowed down enormously the import process. Such things would not be possible if proper IT architecture analysis used first.

Advantages

  1. Company A provides only data and MS SQL Integration Services packages to be run on any MS SQL Server. No program is running on client side non support, no complex application code, no bug fixes or maintenance headache, one app less to maintain/support.
  2. As MS SQL Integration services are designed for high load and performance and data transformation jobs, it is much more reliable and faster.
  3. As the IT architecture concerns the solution is not just “MS SQL Integration services” but any Integrational services including Oracle and DB2. It created an opportunity for a new product for company and taking on new markets. It can be seen as a classic case of IT Architecture delivering business transformational change together with stability, versus a mess of implementing not well fit solution.

Guerrilla tactics employed

  1. Cross team use of expertise. Non SQL team contributed a lot for analysis. While on the first look it seemed to be a requirement only for Java Development team to improve their code, but another team was very happy to change their XML exporter to make it MS SQL Server Integration Services complaint.
  2. Low-cost unconventional approaches were utilized in a specific way suitable for the current solution to achieve the goal:
  1. Using existing MS SQL license with Integration Services against “Java is the best” approach
  2. Hard decision made to ditch “so-long-developed” and “precious” for Java developers program, which was associated with pride and “hard core” programming techniques, “own MS SQL script rendering library”
  3. Quick mockup with MS SQL Integration Services for proof of concept without asking permission from management (“1 day could be potentially wasted!” compared with the reality of more than half a year wasted)
  1. While management agreed and even said “it’s been obvious from the beginning”, some guerrilla warfare took a bit of a place due to resistance from both management (“we have spent so much time already”) and developers (“it works... well, almost, just small fix is required”).

8. Team dynamics analysis

Guerrilla IT Architecture takes team dynamics as a very important area of analysis and action to apply.

Issues found

  1. When a development team makes decisions, a technology choice tends to be based more on their personal technology affection and attachment. Whether it is a desire to learn new technology, to improve their skills or playing with a new toy - often a decision appears to be distant from what is actually needed for a particular case. That’s why a technology choice and overall architecture should not be put fully on developers. In this case the decision making for Java developers was impossible to go near any Microsoft technology. That’s why MS Integration services solution was overlooked.
  2. As the control of development was given to Java team, a non-SQL DB team has not been asked their opinion. They were just given a task to generate a particular XML extract in a specific format. Further analysis showed that non-SQL DB team had a number of warnings regarding a technology choice and also had an interesting architecture solutions.
  3. Java developers considered only a “pure Java” solution and wanted to do their programming due to belief that only Java gives all the goods of hard core and optimized programming. Also the management position was “we have only Java strong skills and programming resources available”. However the solution outside of any programming was missed, very long development time was undertaken and maintenance came to a headache with unhappy clients’ complaints.
  4. An excessive pride in programming, personal achievements and arrogance are the major stoppers of objective thinking. They often come in place where exceeding power of control is given to developers’ level without proper supervision and considering alternative approaches.

Correction

  1. Changed decision making control and allocation of responsibilities. Technology choice advised to be either allocated to a part time or full time architect position or another person who is distanced from development teams and has IT architectural knowledge and experience.
  2. Brought to teams and management awareness the differences between company goals and IT staff personal goals and a possibility of their clash. Learning plans are set up for developers to keep their skills improvement process up and to merge those goals.
  3. Explained view of no reason to split on good/bad technology (MS vs. Java etc).
 

Add comment


Security code
Refresh



Powered by Investor Services. Valid XHTML and CSS.