News and Events

Twister Data Integrator 3.5

The primary goal of the 3.5.0 release was to improve the user's experience when creating data flows with the Graph Designer. The Designer can now display information about the components, ports and connections to help guide the user. To support these enhancements, it was necessary to modify the underlying API that Twister exposes to component developers in order to expose more information to the Designer application. These API changes are not backwards compatible with the previous 3.4 API and thus all components need to be updated to conform to the new requirements.

Component API

1. Access fields by name or index
2. Convenience methods to access field values as IO Streams, XSLT Sources
3. Value conversion is now forced when a field value is populated, instead of when the field value is read.
4. A FLOAT datatype has been added to save memory over using DOUBLE
5. Validation methods have been added to the component API so that each component can define its own validation logic.
6. Support for custom string parsers for Dates, Numbers, Booleans
7. Component labels and help information loaded from XML and HTML files that accompany the component implementation classes.
8. The FieldInfo class has been replaced by FieldMetaData.
9. FieldValueMutators and FieldValueAccessors have been replaced by the Field and FieldLookup classes.
10. RuntimeRecordFormatProvider has been renamed to DynamicRecordFormatProvider.
11. Defined concept of an Execution Phase. The runtime will use this setting in a future release.
12. Added TemplateEngine and Template interfaces. These allow a component to generate text values by merging static and dynamic values.

Runtime Improvements

1. Added SFTP and SCP protocol support using URLs
2. Designer and Runtime using common code for reading and writing projects and data services
3. Loggers share a common configuration

New Components

1. Database Query – Read data from a database using SQL
2. Database Update – Insert/Update/Delete using SQL in small transactions
3. Database Batch – Insert/Update/Delete using SQL in large transactions
4. Datafile Parser – Parse Delimited, Fixed Length or Mixed text data files
5. Document Builder – Create XML documents from a stream of records

Updated Components

1. XPath Parser - Handles XML from source fields of type URL, XML or STRING. Handles mixed content elements differently.
2. Document Transformer - Handles XML from source fields of type URL, XML or STRING.
3. Database Lookup - Shares code with Database Query.
4. Field Filter - Configuration changed.

Controller Updates

1. Process History now displays calendar control for selecting date
2. Users/Roles moved out of tomcat-users.xml file.
3. Passwords now stored as an MD5 hash
4. Administrator is now prompted for password during the setup process, there is no default password.
5. SSL required between nodes and controller
6. Form based authentication for web pages. Basic or x509 authentication for services.
7. Configuration workflow prompts the user to provide a license file.