Machine Learning Arch

What are the steps that the whole process must take to be successful?

1. Connect to CoreDb2 (geo-replication)
2. On a fixed schedule, run logic to prepare any and all csv files needed for machine learning logic.
3. Store the files in a storage.
4. Add the fact that a file was generated to a queue.
5. Have a consumer of the queue (machine learning logic) that runs when a file is ready (train/re-train).
6. When the training is done, save the model as a zip file and store it in storage.
7. Expose the model using api.

Notes

It should be all serverless so that it can scale automatically when needed.

Processing pipeline

1. (cron) Azure Function | azure logic app | storage queue
2. Create event grid events
3. (triggered) Azure function
4. generate data set for each participant
5. store data in blob storage
6. creates event
7. (triggered) Azure function
8. train on new data
9. generated trained model and store in blob storage
10. api loads appropriate zipped ml model for prediction

At this point, azure api endpoint should be able to pick up the latest model and use that to serve predictions.

Event topics

We need event topics to register different publishers and consumers of events in that topic.

Topic	Publisher	Consumer	Number of Events	End Product
INSIGHT-INIT-ML-PIPELINE-DEV	Azure Function (timer triggered)	Azure Function	Number of participants	training data for each ml model(blob storage)
INSIGHT-PROCESS-PARTICIPANT-DATA-DEV	Blob storage	Azure Function	Number of files added to blob storage	Zipped ML Model(blob storage)