Architecting Marketing Cloud at scale – 10 items for your checklist


The human animal differs from the lesser primates in his passion for lists. ~ H. Allen Smith
No matter how expert you may be, well-designed checklists can improve outcomes. ~ Steven Levitt
The checklist is one of the most high powered productivity tool ever discovered. ~ Brian Tracy

Enough on why we need checklists :). Here goes mine.

1. Contact Model:

In modern times, getting data is no more the challenge, getting too much of it is the new challenge. How do you make sense of the data? Do you understand how they are related? Which signals to use for segmentation and campaigns? This is where the contact model comes in. Simply put, this is a set of Data Extensions that are related to a key value, and they show how the subscriber is represented in Marketing Cloud.

2. Sandboxes:

Marketing cloud does not have sandboxes. We do not have a full copy or partial copy sandboxes, which can be promoted to a production instance. Just to make our life a bit easier, what we can do is to procure multiple business units. One Parent BU and have two BU’s within it, one can be used as a prod, and the other one can be QA. A bit of planning beforehand goes a long way. How to implement new changes? How to test email sends? Journeys? Early 2019 Salesforce introduced Deployment packages for Marketing Cloud. So, technically with shared email and Data extensions, we can copy that nice fancy journey we created to our production box with point-click.

3. Data Integration

Getting the data is no longer a technical challenge. Still, we should plan for it. From the point of creation to the data being available for a campaign, how long does it take? What happens if someone deleted their account? There are several ways to integrate data into Marketing cloud. File-based SFTP uploads. Mulesoft, Dell Boomi, Informatica based API integration. Data volume, east of change, cost are some of the criteria that should be considered for the integration,

4. Primary Keys

Data extensions are the main drivers in Marketing cloud. They store subscriber data, provide data to emails, journeys, mobile studio. We use them to create the contact model. One point that often gets overlooked is the ability to create primary keys for Data Extensions. Imagine a data extension with 25M records which stores a coupon code and you wanted to search for a specific coupon code for a subscriber in CA with login name XYZ. Without running a SQL query or filter, Marketing cloud does not provide many alternatives. Here, the primary keys come real handy. If my data extension had user login and state as a composite primary key. I can simply go to Contact builder and search for that user login or state. They also help in long-running queries.

5. SQL Query timeout

A question, I often encounter is what is the size limit of a Data Extension, Salesforce recommends retention and sizing, but there is no hard limit. I have seen data extensions upward of 200M without any specific issues. Having said that, running queries on those data extensions in conjunction with system data views like _sent, _bounce is a completely different ballgame. Before creating a SQL query, ask yourself, why do you need it? Out of the box reports, discover reports may be good alternatives. Data extracts from the Marketing cloud can be scheduled. If SQL queries are the only solution, then remember to break them up by dates or other criteria. 29 minutes is all it takes for Marketing cloud to say that it’s not going to run that query anymore.

6. Optimize All Subscriber view

This is a small task yet important. Salesforce support can optimize the all subscriber view, so it shows up subscribers when the list is huge. The default behavior is to render all the subscribers, for an account which exceed 100M, it means nothing ever shows up.

7. GDPR Compliance

How you would grow your data. Will marketers create new Data Extensions for sends? If user record is deleted, how should you remove the data from the Marketing cloud instance? Think of an account which has grown over the years, multiple marketing teams using it. Thousands of data extensions and then you are hit with GDPR. What do you do? I don’t know. What I know is we can set processes around how do we create new data. In the ideal world, the data sets should come from the integrations and within Marketing cloud, it should be segmented through filters and queries. Once a user opts out, unsubscribe should follow. If there is an external system which deletes user records, that also should have a way to sync with Marketing Cloud. Contact deletion is possible, It can only happen from parent BU and 1M at a time. More importantly, start with a plan on how to delete and archive data.

8. Verification of Processes

When we are talking about multiple data ingestions and millions of records each day, a lot of things can go wrong. Build process checks. Automation studio provides an activity named verification. If you are expecting an hourly feed of active users and it varies between 100K to a million, You can create a Data Extension size check through the activity and if it’s ever less than 50K, people who need to know to get the alerts. Similarly, create queries for data extension which are expected to grow over time. kind of an audit log.

9. IP Pools

I am assuming you have already warmed at least one IP. If we are talking about less than 1M sends, we are all good. If you envision your account to grow 10X. Maybe 20M sends in a day? Keeping an IP pool comes in handy, remember each IP will take 4-6 weeks to warm up. I will bucket 2M sends per IP.

10. Reporting and Analytics

Marketing cloud provides three ways to provide reporting data, out of the box reports which you configure, create extraction and push them to an FTP folder, run your own queries and filters and store the data in Data Extensions. Think which requires the least customization and least maintenance.

A bonus one

AMPScript and SSJS are powerful tools. They can create, read from, write to Data Extensions. Before designing a system which uses them, please keep in mind they have data retrieve limits of 2000 and 2500 records. SSJS offers ways to retrieve more records and you can also devise your own ways, something to keep in mind when you are designing the system processes.