Performance of bc2adls
In my deep dive session on Areopa Webinars (Areopa – Dynamics 365 Business Central Expert Training) I had told that I will do a performance test on the bc2adls (microsoft/bc2adls: Exporting data from Dynamics 365 Business Central to Azure data lake storage (github.com)) extension.
BTW the deep dive session you can find here:
20220426 – Sync your Business Central data with Azure Data Lake – YouTube
The test setup:
I have created a setup that export 107 tables from Business Central to Azure Data Lake.
Also my Azure Synapse runs on the default.
I have added the Performance Toolkit to generate allot of data and runned it three times to measure any differences.
The first initial run exported 42 (there were some errors in the tables) tables with in total 72.201 records. The total time for exporting the records to the Azure Data lake took 2 minutes.
The total time in Azure Synapse took up to 03:49.
The second run exported 55 tables with in total 303.481 records.
The total time for exporting the records to the Azure Data lake took 9 minutes.
In this run I had a couple of deadlocks on tables:
The activity was deadlocked with another user who was modifying the ADLSE Current Session table. Please retry the activity.
"ADLSE Current Session"(Table 82565).IsAnySessionActiveForOtherExports line 3 - Azure Data Lake Storage Export 4PS by 4PS B.V.
"ADLSE Execute"(CodeUnit 82561).TrySetStateFinished line 8 - Azure Data Lake Storage Export 4PS by 4PS B.V.
"ADLSE Execute"(CodeUnit 82561).SetStateFinished line 2 - Azure Data Lake Storage Export 4PS by 4PS B.V.
"ADLSE Execute"(CodeUnit 82561).OnRun(Trigger) line 88 - Azure Data Lake Storage Export 4PS by 4PS B.V.
and
The activity was deadlocked with another user who was modifying the ADLSE Setup table. Please retry the activity.
"ADLSE Execute"(CodeUnit 82561).AcquireLockonADLSESetup line 3 - Azure Data Lake Storage Export 4PS by 4PS B.V.
"ADLSE Execute"(CodeUnit 82561).TrySetStateFinished line 6 - Azure Data Lake Storage Export 4PS by 4PS B.V.
"ADLSE Execute"(CodeUnit 82561).SetStateFinished line 2 - Azure Data Lake Storage Export 4PS by 4PS B.V.
"ADLSE Execute"(CodeUnit 82561).OnRun(Trigger) line 88 - Azure Data Lake Storage Export 4PS by 4PS B.V.
I run this export three times. So the 9 minutes is a combined calculation.
The total time in Azure Synapse took up to 05:40.
The third and last run the export was 107 tables with in total 419.824 records.
The total time for exporting the records to the Azure Data lake took 8 minutes.
The total time in Azure Synapse took up to 06:15.
In a short overview:
# | Total tables | Total records | Duration BC | Duration Synapse |
1 | 42 | 72.201 | 2 minutes | 03:49 |
2 | 55 | 303.481 | 9 minutes | 05:40 |
3 | 107 | 419.824 | 8 minutes | 06:15 |
Conclusions
Well overall the export to Azure Data Lake from Business Central was very impressive. There were some deadlocks in it. But they are resolved by adding a commit:
I have done it before the third run and after that I hadn’t any deadlocks anymore.
My only concerns is the duration of Azure Synapse. I have done it with the standard configuration. So maybe you can tweak that (Pricing – Azure Synapse Analytics | Microsoft Azure). Also you can run in over night
But overall got a good feeling that this will perform very well.
Below is the detailled information about the performance test.
5 COMMENTS