DataImporter - Import Data into Octopus

Table of contents

Introduction

The DataImporter program (ESI.Octopus.DataImporterApp.exe) can be used to import and/or synchronize data from an external source that is compatible with OLEDB/ODBC. 

DataImporter can be used to perform the initial import of data or to synchronize on a regular basis data contained in another system.

This article explains how DataImporter works and the different types of import available.


 

References

Links to the articles related to the import sources and the XML file. 

 

Prerequisites

  • Create an Octopus system account that will be used to run the task. See article Octopus System Account.
  • XML file containing the information on the source of the data to be imported. See article DataImporter - XML Configuration File .
  • Presence on the workstation/server running the DataImporter task of an OLEDB provider compatible with the data source to be imported (the ODBC provider is generally pre-installed).
  • Installing a 32-bits version of Microsoft Office on the workstation/server running DataImporter.
  • Data source configured as specified in the section Data Source Specifications for Import

How it works

DataImporter makes it possible to import different elements in Octopus (Users, CI, Suppliers, etc.) from a OLEDB/ODBC data source (CSV, Excel, Access, SQL Server, Oracle, MySQL, etc.).

  • DataImporter can replace the ADSIReader program when the users are not in Active Directory or when the a data source is more complete elsewhere.
  • DataImporter can be used to complete the information gathered by WMIUpdater (Order Number, Purchase Price, Date of Purchase, etc.) or to import information from non-Windows devices (Printer, Switch, Projector, etc.) or even from Linux servers or Mac computers. 

The 3 essential elements to operating the DataImporter program are:

  • Data Source (Excel file, Access table, SQL view, etc.)
  • XML File; this file indicates the data source, where it can be found and the view/table to import.
  • Command Line; to run the program..
NOTE : DataImporter is used to add or update the Octopus CMDB with the available data source. But it cannot be used to delete records or erase fields, with the exception of the EmptyValueHandling tag. 
The exceptions are mentioned directly in the tag section of the article on import sources,

How to Use DataImporter

DataImporter is a Command Line (DOS) type of program. It is conceived to be easy to automate

  • The complete name of the program is: ESI.Octopus.DataImporterApp.exe
  • It is located in the Octopus folder (C:\Program Files\Octopus) or in (\%homepath%\AppData\Local\Octopus).
  • Example of folders where Octopus can be located:
    • For Windows 7: C:\Users\slevesque\AppData\Local\Octopus
    • For Windows XP: C:\Documents and Settings\Slevesque\Local Settings\Application Data\Octopus

Parameters

There are 6 available parameters, of which 4 are mandatory, to execute DataImporter:

Mandatory Parameters

  • /Login: Octopus Username
  • /Password: Octopus Password
  • /ConfigFilePath: Path to the DataImporter configuration file
  • /Team (from version 3.8 only)

    • The number of the team to be updated. By default, the IT team is 1

Example: ESI.Octopus.DataImporterApp.exe /Login:system /Password:octo /ConfigFilePath:c:\Import\CI.xml /team:1

Optional Parameters

  • /LogFilePath: To move the location of the log file. By default, the file is named DataImporter.log and will be located in the Octopus folder. If you move the file to another folder, make sure this folder exists (the system cannot create the folder). Furthermore the path must contain the name of the file 

Example : /LogFilePath:C:\LogFiles\ImportJournal.log

  • /Debug: Our Service Centre may ask you to execute the command line with the /Debug parameter in cases of problems with DataImporter. This parameter generates a binary file that enables us to analyze the errors when we do not have access to the imported data. The generated file can then be used with DataImporter as follows:/UseDebugOutput:FileName.
  • /ExtractionTimeout TimeOutInSeconds:  Sometimes the data source providing data to Octopus will be unable to complete within the allocated five minutes.  In which case, you can use the ExtractionTimeout parameter to specify how many seconds the extraction is allowed to run.

Automate DataImporter

Refer to the Windows Scheduled Task article to find out how to automate imports with DataImporter.

Configuration File (XML)

The configuration file is an XML format text file that contains all the information about the data source to import. Refer to the following article explaining how to program the references of your data source. 

Troubleshooting

This section describes the ways to help you troubleshoot issues.

Operation Log File

During execution, DataImporter creates a log file in addition to the results returned on screen. This log is saved in the DataImporter.log file and it is located in the folder in which DataImporter is run, the Octopus folder.

Consult this file after each import, as it lists all the error messages of the data that was not imported.
 

What you need to know:
Each time DataImporter runs, the size of the file is controlled. If it exceeds 50K, DataImporter will rename the current file in the following format DataImporter_YYYYMMDD_hhmmss.log.

Error Message Analysis

When the import is successful, the following lines will be in the log file:

05-03-13 11:36:16 - *****************************
05-03-13 11:36:16 - Reading source <[XXXXX$]>.
05-03-13 11:36:17 - Importing source <[XXXXX$]>.
05-03-13 11:36:18 - Import from source <[XXXXX$]> successful. n record(s) imported.
05-03-13 11:36:18 -
05-03-13 11:36:18 - Completed.
  • The source <XXXXX> Represents the name of the view or of the tab being imported
  • n record(s) indicates the number of recordings successfully imported

However, when there are one of more errors, the following lines will be in the log file:


05-03-13 11:37:55 - *****************************
05-03-13 11:37:55 - Reading source <[XXXXX$]>.
05-03-13 11:37:55 - Importing source <[XXXXX$]>.
05-03-13 11:37:56 - Error importing source <[XXXXX$]> : import of n record(s) failed, n record(s) imported successfully.

Here is a list of possible error message and how to correct them.

NOTE : When importing data, it is important to know about the following behavior:

  • When an error is found on a line, the whole line gets rejected and none of the fields will be imported.
  • When many errors are found on the same line, DataImporter will only show the first one. So more then one import would be needed to find all the errors and correct them.
  • ServiceProxy : unknown service infrastructure exception error occurred

    • DataImporter is unable to connect to the server.

    • Make sure you are able to start the Octopus client to confirm it is not corrupted. If required, download a fresh version of the client.

  • An unexpected error has occurred: The 'xxxxxx' provider is not registered on the local machine.

    • The OLEDB provider associated with the data source is not installed on the machine running DataImporter.

    • Make sure you have a 32-bits version of Microsoft Office.

  • An unexpected error has occurred: Invalid character in the given encoding. Line X, position Y.

    • This error is related to a mistake in the XML file. Please refer to the next section to configure the XML file. Make sure there are no accents in the name of the source.

  • An unexpected error has occurred: '.', hexadecimal value 0x00, is an invalid character. Line 2, position 1.

    • The XML file has been saved in Unicode instead of ANSI.

  • Cannot import this record because of an irresolvable naming conflict.

    • The record that is being imported or updated is not unique. The system does not know which one to use.

  • XX$ : XXXX : RequiredColumn

    • Indicates that a mandatory column is missing.

  • The data modified is in read-only mode

    • The Octopus User used in the command line does not have all the required permissions to run this type of import.

  • The specified main contact does not exist

    • Make sure that the Main Contact used exists and that the correct Windows Username is used during the import.

  • Some fields contain invalid values (CI Import)

    • TypeName: The value specified in the 'Type' field is invalid.

    • PurchasePrice: The value in the 'PurchasePrice' field must be numerical.

    • Status: The value specified in the 'Status' field is invalid.

  • There are no attributes called XXX for this CI type (CI Import)

    • When the system does not recognize the name of a column, it will suppose that it is the name of a non-existing or misspelled attribute. 

    • If the name really exist, make sure that it is properly spelled and that there are no spaces in the name in the Excel cell.

  • The attribute XXXXX is required

    • A mandatory column is missing in the data source.

  • There are no attributes called 'Test#1'' for this CI type.

    • If the name of an attribute contains a period (.), the system cannot interpret it and will replace it with the pound sign (#) in the error message of the log. So there should not be any periods in the names of attributes.

  • An unexpected error has occurred: String was not recognized as a valid TimeSpan

    • When importing the Effort field in requests, the information must be in the following format: 03:45 or 00:30. 

  • The value specified for field 'XXXX' is invalid.

    • The value that is being imported is not part of the accepted values for this field type. 

  • An unexpected error has occurred: Could not find file 'C:\Import\XYZ.xml'

    • The colon (:) was missing from between the parameter and it's value in the command line.

​If you encounter an unknown error message, please Contact us.

X
Help us improve our articles








Help us improve our articles