Enterprise App Data Portability
How do you take a lot of data from a large relational database, make it easy to import & export, and make it so that it isn't invalidated each time you change your schema or release a new application version? How do you handle importing older data? How do you handling importing multiple very different data types? This was an interesting problem. I initially "cheated" on this. I didn't really solve it. Instead, when we were on a document database, I just created a blank temporary database for each import and overlayed the imported data on it, then reran all the migrations (yay idempotency). This worked and was easy to implement, but felt kind of gross. Once we moved to a relational DB things got more interesting. Building a temporary DB and running migrations came with a lot of drawbacks so instead we did a baseline schema to class dump and then schema diff on each build. Any time the schema that is used in data import / export changes, we write a new versioned class file for the schema. This allows us to keep a baseline behavior for import and override it when there are changes to behavior, new fields, or some form of migration in code that's needed to handle the import process. This also underpins a more user-friendly content creation process in the API which makes it much faster to get data into the system for users so they don't need to know the details of the entire data model.
| Type | Data portability |
| Operations | Import / export / migrate |
| Property | Resumable, idempotent |
| Span | 2019-2024 |
| Visibility | Work (proprietary) |
| Source | Closed |