This Data firehose developer reference covers everything from handling sync errors and setting up authentication to understanding the data schema and IP whitelisting requirements.

Error handling

Effective error handling is crucial for maintaining the reliability and efficiency of the data firehose. This section outlines how sync errors are managed, the notification options available, and the retry mechanism in place.

View sync errors

Sync status - quick overview
Sync errors are shown under the sync status on the Data firehose settings page. This gives you a quick overview of any issues.

To get to the Data firehose page, in Dotdigital, go to Connect > Data firehose.

Sync status - detailed error report

The sync report page also shows sync errors as Failed. However you can view more detailed information about the failure.

To view the detailed status report:

In Dotdigital, go to Connect > Data firehose.
Find the Data firehose you want to view the report for, then select Report.
Under Sync details, select Failed in the Status column. This open the report sidebar.

Sync error notifications

You can set up notifications for sync failures over email, in-app notification, or both, ensuring you are quickly informed of any issues. Notifications are triggered every time a failure occurs.

You can choose the notification method when you set up your Data firehose. However, to edit your notification method after setup:

In Dotdigital, go to Connect > Data firehose.
Find the Data firehose you want to edit, then select Edit.
Select NEXT until you get to Step 3: Frequency.
Expand the Sync failure notification drop-down menu, then choose the notification method.
Options:
- None
- Email
- In-app notification
- Email and in-app notification
If asked, enter your email address.
Select NEXT, and then UPDATE SYNC.

Retry mechanism

If a sync encounters an error, it is interrupted and marked as failed. A new sync is then scheduled at intervals of 15 minutes, 1 hour, or 1 day, depending on the configuration. If successful, the next sync processes all of the un-synced events, to make sure no data is lost.

There is no threshold for the maximum number of failed syncs. The system continues attempting to sync at the scheduled interval.

Types of errors

Understanding the types of errors that can occur during the sync process is important for effective troubleshooting.

A failed sync error is the generic term for when the synchronisation between Dotdigital and the chosen data storage fails. There are various reasons for this, known as error types, which are categorised below.

Error type	Description
Authentication error	These errors occur when there are changes since the firehose sync setup, such as user removal or password changes. Make sure your authentication credentials are up-to-date to help prevent these errors.
Permissions error	Permissions errors happen when the necessary permissions, such as directory creation, are not checked during configuration. For example, if the directory creation permission is not granted, the sync process fails when attempting to create directories.
Connection error	Connection errors are typically caused by transient network issues or the availability of the data warehouse service. These errors are often temporary and might resolve themselves, but persistent issues might need further investigation.

Authentication processes

Proper authentication is essential for securely connecting to your data storage solution. For detailed instructions on setting up authentication with SFTP/FTP and Amazon S3, see the article Set up a data firehose.

User permissions

Using the correct user permissions is important for a smooth and secure Data firehose sync.

SFTP/FTPS user permissions

You must have permission to write to and create directories. We can help in creating directories based on connection details.

Amazon S3 permissions

You must have permission to write data to the destination.

IP address and whitelisting

You must make sure the correct IP addresses are whitelisted to maintain a secure and reliable firehose connection. Learn more in IP addresses used by Dotdigital.

Data schema

Understanding the structure of the data being synced is essential for effective integration and troubleshooting.

Learn more in Dotdigital Data firehose event schema.