This data firehose developer reference covers everything from handling sync errors and setting up authentication to understanding the data schema and IP safelisting requirements.

Overview

Data Firehose is a powerful tool that allows you to sync events from Dotdigital to various external data storage platforms or warehouses.

We support syncing data to:

Google Cloud Storage
Azure Blob Storage
Amazon S3
SFTP
FTPS

Learn more about getting started with Data Firehose here , this article covers everything from handling sync errors and setting up authentication to understanding the data schema and IP safelisting requirements.

Error handling

Effective error handling is crucial for maintaining the reliability and efficiency of the data firehose. This section outlines how sync errors are managed, the notification options available, and the retry mechanism in place.

View sync errors

Sync errors are shown in the Status column on the Data firehose report page in Dotdigital. This gives you a quick overview of any issues.

To access:

In Dotdigital, go to Connect > Data firehose.
Select Report icon for the firehose connection you want to view reporting for.

Sync status - detailed error report

The sync report page shows sync errors as Failed. You can view more detailed information about the failure.

To view the detailed status report, under Sync details, select Failed in the Status column. This opens the report side panel.

Sync error notifications

You can set up notifications for sync failures through email, in-app notification, or both, ensuring you are quickly informed of any issues. Notifications are triggered every time a failure occurs.

You can choose the notification method when you set up your Data firehose. However, to edit your notification method after setup:

In Dotdigital, go to Connect > Data firehose.
Find the data firehose you want, then select Edit.
Select NEXT until you get to Step 3: Frequency.
Expand the Sync failure notification drop-down menu, then choose the notification method.
Options:
- None
- Email
- In-app notification
- Email and in-app notification
If you selected an option including email, enter your email address.
Select NEXT, and then UPDATE SYNC.

Retry mechanism

If a sync encounters an error, it's interrupted and marked as failed. A new sync is then scheduled at intervals of 15 minutes, 1 hour, or 1 day, depending on the configuration. If successful, the next sync processes all of the un-synced events, to make sure no data is lost.

There is no threshold for the maximum number of failed syncs. The system continues attempting to sync at the scheduled interval.

Types of errors

Understanding the types of errors that can occur during the sync process is important for effective troubleshooting.

A failed sync error is the generic term for when the synchronisation between Dotdigital and the chosen data storage fails. There are various reasons for this, known as error types, which are categorised below.

Error type	Description
Authentication error	These errors occur when there are changes since the firehose sync setup, such as user removal or password changes. Make sure your authentication credentials are up-to-date to help prevent these errors.
Permissions error	Permissions errors happen when the necessary permissions, such as directory creation, are not checked during configuration. For example, if the directory creation permission is not granted, the sync process fails when attempting to create directories.
Connection error	Connection errors are typically caused by transient network issues or the availability of the data warehouse service. These errors are often temporary and might resolve themselves, but persistent issues might need further investigation.

Error type

Description

Authentication error

These errors occur when there are changes since the firehose sync setup, such as user removal or password changes.

Make sure your authentication credentials are up-to-date to help prevent these errors.

Permissions error

Permissions errors happen when the necessary permissions, such as directory creation, are not checked during configuration. For example, if the directory creation permission is not granted, the sync process fails when attempting to create directories.

Connection error

Connection errors are typically caused by transient network issues or the availability of the data warehouse service. These errors are often temporary and might resolve themselves, but persistent issues might need further investigation.

Authentication processes

Proper authentication is essential for securely connecting to your data storage solution.

Learn more in Set up a data firehose.

User permissions

Using the correct user permissions is important for a smooth and secure data firehose sync.

SFTP/FTPS user permissions

You must have permission to write to and create directories. We can help in creating directories based on connection details.

Amazon S3 permissions

You must have permission to write data to the destination.

IP address and safelisting

You must make sure the correct IP addresses are safelisted to maintain a secure and reliable firehose connection.

Learn more in IP addresses used by Dotdigital.

Data schema

Understanding the structure of the data being synced is essential for effective integration and troubleshooting.

Learn more in Dotdigital Data firehose event schema.

Data Firehose API

The Data Firehose API can be used to programatically setup Data Firehose configurations as an alternative to manually configuring within the app. This is ideal where Data Firehose configurations may need security credentials updating on a regular schedule or when you have many accounts to administer.

Learn more in Data Firehose API section.