How to model master data / tenant specific models

[pmartelletti]

Hi @codeliner ! I know you guys are busy right now with the fee office example (#7) , but just thought I'd place the question here, explain my domain and then, whenever you guys get a minute to review it, we could go over it. 😄

So, a bit of a context from our company here: we're a company that builds software for grocery stores, most of the hard work is done for the POS in the actual store and in the back office. The POS is built in .NET, and currently being rebuilt using CQRS.

My team, however, is focused more in web applications that support our customers, like transferring files from store to head office, place automatic orders, sync products and offers between the different head offices (as the main products base and offers is the same for all customers). We use PHP (mostly Laravel) to build our apps, and JS to connect to the API's we generate.

The project we're currently working is the offers one. This is already working, the customers are happy with it. However, a new requirement was added to the system, and things started to get messy: because we didn't have a log of what was happening and when in the system, we were unable to respond questions from the customers like "when was this item introduced to this offer" or "when was this offer extended". That's when we decided to stop what we're doing a try to come up with some ideas on how we could implement this. Use Laravel log could have been enough for the actual questions/reports we have, but that wouldn't help us for any future questions customers will have. That's why, after hearing the other team talking about ES all the time, we decided to give it a shot.

=======

Now, some background on the Domain and what our app needs to do.

All our customers receive daily a text file saying which offers are active (or will be), and what items are in that offer (they are all part of a cooperative group, so basically all members of the group receive the same offers, as the group makes the purchase for them). If they use our EPOS system, then we offer this added service where we parse the file and sync their Head Office with the deals, automatically - so no human interaction is needed. If they are not using our system, then they'd have to manually key in the changes daily.

The domain was easy enough when all customers receive all the same deals - however, it started getting complicated when the new requirement was added: customers would like to be able to ignore deals they are not interested to offer, to create their own deals, or to amend deals (that is, if the deal was buy any soft drink for €1, but the group said only Coca-Cola brand was in the offer, there might be a store that wants to add Pepsi brand as well).

That's were it all got complicated - we haven't found so far any Domain that talks about "Master Data" and "Tenant" specific data. The aggregate started to look really ugly when we added flags like "this offer is specific to this tenant", or "this tenant is ignoring this or that offer".

So here's where we can to your help. Can you help us try to get the correct models for our domain? Here's a bit of example of what the system should be doing.

• Be able to import offers, that have a set of properties (like, description, start date, end date, reward, and mechanism - when an offer should be triggered).
• Be able to add different groups to the offer. Most of the offers will only have one group (ie, buy 2 from this group and pay $3). However, there are offers that need to have more than one, for example: buy one of each group and pay $3 (meal deals, for example, where you would buy a soft drink, sandwich and dessert).
• Be able to add/remove items from the groups.

That's essentially what we had before. The aggregates were easy to understand, it was all straightforward.

However, the things got more complicated with these new requirements, as customers should be able to say:

• ignore any offer containing this product
• ignore this particular offer
• I want to edit this offer (so, change some details of it - like adding new items) but I still want to get updates from the master data
• I want to edit this offer but stop getting updates from master data
• I want to create a new offer that is only available for my stores, and if a deal from the master data comes in with the same details (start date, end data and deal number are used to identify an offer), then ignore it and use mine.

So, with all these new rules, it all started to get complicated and we're kind of stuck, or slowly making any progress. So wanted to know if you could help us:

a) Try to understand how tenant data would work, as I've never seen an example for DDD like that
b) How would you handle the ignores or the custom data? Where would you keep track of that?, etc

Also, we really need to have a track of the import process, what was imported and when, and what failed, what was skipped because of the customer settings, etc, as we usually get questions like "why is this item not in my deal, but it is in the other customer?". Would you do that in DDD as well?

Thanks a lot for your help in advance - and hope the problem is clear. I'll more than glad to help clear out any questions you may have about it.

[codeliner]

Hey @pmartelletti , very interesting problem space ;) thx for sharing.

You're right. We're really busy at the moment. We joined a developer team to help them with their core system. It's an enterprise business and they are currently in trouble so we have a lot of work on our desk. We need to understand the system, identify bottlenecks and remove them. Then we need to split the system into smaller units and stabilize it.
This means that the next months (probably until February/March 2019) we can't spend much time for OSS. My plan is to find a bit time every week to continue with the Fee Office. We're working on it since a few weeks and collected a lot of information, but @enumag 's team still needs some answers/guidance.

That said, I don't think we can go through your problem this year, sorry. But maybe some other ppl in the prooph community can help. ping @gquemener ?

Anyway, you can read through the Fee Office issue and see what we did there. I'm currently summarizing the issue here: https://proophsoftware.github.io/fee-office/ddd/

In short:

• we analyzed the domain
• created a context map
• event storming
• protoype implementation (wip) with Event Machine: https://github.com/proophsoftware/event-machine

Do an event storming session with your team and share the result on public board on https://realtimeboard.com/

Do the Event Machine Tutorial: https://proophsoftware.github.io/event-machine/tutorial/ (takes 2-4 hours)
and start working on a prototype using the Skeleton: https://github.com/proophsoftware/event-machine-skeleton

Make it a public repo and invite ppl to help you with the design. I always have 15 min during a break or so to give some advice, but the next emergency-call has to wait.

[pmartelletti]

hey @codeliner - thanks for your reply! Yeah, I suspected that. Because we do have a deadline, we started working on the code, even we didn't like how it turned. Most probably the end result will be fine, and will work, however, we still feel there's stuff that can be improved so that's why I came to ask for help, maybe for a second iteration of the development (or some refactoring!).

So, what I can do is work during this weekend on a prototype using the Skeleton, make it public and share it here, and maybe when you have 10/15 minutes to have a look, you or any other member of the community could tell us if we're going in the right way, or if we're really complicating ourselves too much and things could have done easier / better.

Does that sounds good to you?

Thanks again!

[codeliner]

One advice for your real world project:
Try to set up a module system similar to the one I've explained here: https://proophsoftware.github.io/fee-office/intro/module_system.html#1-3

Not sure how to do it with Laravel, but I'm pretty sure that it is possible. When you have those modules and follow the rules then it's easier later to throw away some of them and replace with a better implementation. Greg Young did a great talk about that topic: https://vimeo.com/108441214
Try to design the system in a way that you can easily delete code. That's a secret weapon to fight the unknown unknowns

[enumag]

This is quite interesting topic. I'll definitely keep an eye on it because we will need something similar in our system as well eventually.

@pmartelletti As you already know I'm not an expert but let me share my thoughts anyway. I think I'd go with something like keeping the master offers as templates only. Each customer would have a set of rules how to use the master offers. Then there would be some ImportOffersProcess aggregate that would take the master offers and customer rules to create customer offers (for one customer). The process would keep a track of what it did with each master offer - if it got changed or ignored and which rule was used to do that, answering your questions "why is this item not in my deal". Is this something close to how you're approaching it? What are the problems with this approach?

[pmartelletti]

hi @enumag thanks for your interest in our case 😄

Well, to be honest, we though about something like that at some point, but we would have ended up with 2 aggregates that were identical, one for the master data, and one for the specific / tenant offers. So we decided to have one OfferAggregate, with an optional flag saying if that was tenant-specific offer (a tenantId, basically), or if it applied to everyone.

Then, on the read model, we'd generate an entry per tenant for that given offer (if the offer was for everyone), skipping the tenants that had any rule to ignore it.

So basically, the 'ignore' part was done on the projection, which I'm not sure if it's 100% correct, thus why I came here asking for help 😆

Makes a bit of sense what we're doing?

[gquemener]

Hello 👋

In term of responsibility, I believe the Mother Company will always be the one sending offers on a daily basis to the customers.
It's up to the customers to ignore or not these offers, based on their own sets of rules.

So I would consider a customer (is this how you refer to them in your company?) has an aggregate gathering, among other things, a set of rules which will dynamically define which offer has to be accepted or rejected (just inventing the terms, again the question is how this process is referred to from your business expert perspective).

What do you think?

[pmartelletti]

@gquemener thanks for joining the conversation 😄

Yup, that's exactly what we have for now. The question is more: how do we store the offers? Should we keep a record of all the offers that were sent from the Mother Company (in case, I don't know, a new customer is added?)? Should we just keep Offers that the Companies accepted (with a reference to the company id in that aggregate?) Should they be 2 diff aggregates?

That's what our main issue here. How to avoid duplication by creating 2 diff aggregates that hold the same properties, but to the business, they represent 2 different things: one is a "template", one is the actual offer that will be sent to the customer.

Makes more sense? 😄

[gquemener]

Yep it does.

Although they look similar, they are not the same thing AFAIU.

I kind of see two contexts, a "Mother Company" context in which an offer represents something, and a "Customer" context, in which an offer represents the same something, plus the fact it's been accepted/process/handled or rejected/ignored (the exact term is the one that's used by your customer!).

It's important to see that this two aggregates will have their own separate consistency boundaries. If you were to reference a mother company offer within a customer offer, then the acceptance/rejection of the later will lose its sense/consistency if the mother company modifies the content of the offer. That's why I think it is important to duplicate this information.

As you say, a Mother Company Offer is a template, which solely is useful for Customer Offer creation purpose.

Should we just keep Offers that the Companies accepted

Which term do the experts use? Company, Customer, Client?

[pmartelletti]

Sorry about the mixup, we use customers. So basically, we have templates for the Mother Company, and actual offers accepted / rejected by our customers.

So, how would you handle duplication then? Let's say I add a new property to the template aggregate - that would mean I'd have to manually copy that details in the Customer Offer aggregate as well, Including business checks?

I can't think of the Mother Template being a simplified version of the Customer Offer, as the checks for doing actions (adding items, adding groups, change start dates, etc) should be the same in both. That's what worries me the most - having to duplicate all these domain logic that's inside the aggregate and, if they are in 2 separate bounded context, it would be duplicated (and there's a potential issue of forget adding / removing / changing in one of them 😟 )

[gquemener]

In my understanding, the template is solely used at customer offer creation time. Subsequent template modification wouldn't impact already existing customer offers, but I may have a bad understanding of your domain.

I think this is the time when an Event Storming session might become handy.
Are you familiar with the concept?

[gquemener]

and regarding offer implementation, I would use something close to the EAV pattern (which allows to add any number of attributes with different values to a given entity/aggregate) => Open to extension / close to modification. But it'll depend on your business requirements :)

[pmartelletti]

In my understanding, the template is solely used at customer offer creation time. Subsequent template modification wouldn't impact already existing customer offers, but I may have a bad understanding of your domain.

That's the problem. If the template receives an update, then it should spread to all the customers that in their settings want to keep the offer in "sync". By default, this is the case - so it's the most common scenario. A modification in the template wouldn't affect the Customer Offer if they've done a modification to the offer, for example. There are some other rules (posted in my first comment), but I think you get the idea now.

We did an Event Storming before actually starting to implement this. And we came up with this model, but to be honest, I was the only in the team with knowledge on the domain and Event Storm, so basically, I could have "contaminated" that Event Storming session with my initial implementation ideas. 😟

[gquemener]

The context map is pretty straightforward in my head:

 +----------------+          +----------+
 |                |u        d|          |
 | Mother Company +----------+ Customer |
 |                |          |          |
 +----------------+          +----------+

PS: Please note that I'm quite experiencing and learning here, so don't take everything I say as correct! Let it be a start for further discussion :)

[gquemener]

so basically, I could have "contaminated" that Event Storming session with my initial implementation ideas.

That happens often, and it's normal as we've been taught to think like that! Do not worry about implementation, it will naturally flow from the domain model ;)

Let's take the TemplatePropertyAdded event. If this event needs to be propagated through Customer Offers that match some conditions, then I would produce this exact message to the outside world (meaning outside of the Mother Company bounded context) through a message queue for instance. Then there would be one (possibly scaled horizontally) consumer on the "Customer" side that would propagate the change given its own set of rules.

WDYT?