Introduction
Microsoft remains steadfast in its dedication to modernize its services, striving to enhance service reliability and efficiency through thorough testing. This discussion will delve into the utilization of Azure Load Testing (ALT) to strengthen the reliability of both Microsoft Fabric and Azure Synapse.
Azure Synapse Analytics represents a cutting-edge analytics service facilitating rapid insights across data warehouses and big data systems. It amalgamates the best features of SQL technologies employed in enterprise data warehousing, Apache Spark technologies for big data processing, and Azure Data Explorer for log and time series analytics.
Microsoft Fabric stands as a comprehensive analytics solution tailored for enterprises, encompassing a broad spectrum from data movement to data science, real-time analytics, and business intelligence. It offers a diverse range of services including data lake, data engineering, and data integration, all conveniently accessible under one umbrella. Both Azure Synapse and Microsoft Fabric leverage the same foundational architecture for data querying, ensuring seamless integration and interoperability.
Objective
The primary aim is to thoroughly stress-test the SQL analytics runtime of both Microsoft Fabric and Azure Synapse, meticulously scrutinizing both server-side and client-side APIs to identify and address any potential issues. The focus lies on detecting and resolving potential problems within the product, making load testing an indispensable part of the sign-off process, necessitating daily stress tests of the system. The ultimate goal is to ensure that a specific code version, when deployed in a particular environment, operates in accordance with the highest standards, thus guaranteeing the best possible user experience.
Requirements
Selecting the appropriate tool for this task was paramount. The list of requirements encompassed:
1. Capacity for Concurrent Query Execution: The tool should be capable of executing a substantial number of simultaneous queries within a defined timeframe, ranging from several thousand to several hundred thousand.
2. Parameterization Flexibility: It should facilitate parameterization to allow for easy adjustment of load intensity, test endpoints, and overall test behavior.
3. Result Validation and Telemetry Generation: The system should validate test results and enable the generation of client telemetry, allowing for correlation with server telemetry.
4. Azure Integration: It must utilize Azure for test execution and scalability, seamlessly run tests from Azure DevOps pipelines, store test definitions in a source control repository, and ensure secure execution without compromising sensitive information.
5. Customizability: The tool should be extensible with custom-written components to cater to specific requirements.
Azure Load Testing (ALT) met all these criteria. ALT is a fully managed load-testing service that facilitates high-scale load generation. It provides robust support for JMeter, a widely used open-source software designed for performance assessment and load testing of applications. Leveraging JMeter, necessary queries and API calls for testing could be seamlessly executed. The service seamlessly integrates with Azure Repos and Azure Pipelines, supports Azure Key Vault for secure storage of secrets, and allows the incorporation of custom code to augment functionality. Notably, the consistency in experience between running tests locally and using Azure Load Testing is remarkable, requiring no modifications to the test script (jmx).
Approach
Test Scenario
The assessment focuses on evaluating the resilience of Microsoft Fabric and Azure Synapse when subjected to high-stress conditions involving varying data sizes, queries, scenarios, and environments. Key processes such as workspace creation, data loading, query execution, and workspace deletion are meticulously scrutinized. To ensure comprehensive evaluation across diverse conditions, tests are conducted on a daily and weekly basis.
Organization of the Testing Project
Each new test cycle begins on a developer’s workstation, where they define the specific scenarios to be tested. Individual test suites are organized into separate folders within a designated repository. These folders contain jmx files, which are executable both locally and through pipelines. Detailed specifications regarding test execution are outlined in ALT YAML files and JMeter properties files. These documents elaborate on various test parameters including:
• The environment in which the test will be conducted (e.g., test, stage, prod)
• The dataset or database being utilized
• The number of simultaneous users
• Wait times
• Ramp-up time
• Test duration
The JMeter test framework is structured to validate the accuracy of results after each query execution. Upon completion of a test, the developer pushes it to the repository. Approval of a pull request initiates an artifact creation pipeline, responsible for constructing binaries, performing unit tests, and publishing all new files to an artifact storage. Subsequently, these files are made available for the Main Pipeline, which retrieves the artifacts, downloads the necessary Azure Key Vault secrets, and executes the ALT tests. Following the test’s conclusion, client telemetry is transmitted to a dedicated Azure Synapse Analytics workspace. This workspace processes the incoming user data, integrates it with client telemetry, and ultimately presents the consolidated data for comprehensive analysis and visualization through Power BI.
Tests have been devised in JMeter for both Microsoft Fabric and Azure Synapse. The flexibility of JMeter allows for script customization to meet specific requirements. The same test script has been reused across numerous scenarios utilizing JMeter user properties. Additionally, custom samplers for JMeter have been developed to address specific needs such as targeting loads, extracting client telemetry, and modifying the internal state of the test environment and services. Notably, Azure Load Testing provides a secure environment for executing custom samplers, ensuring the integrity and confidentiality of test operations.
Automated Test Execution
All documentation outlining test plans is securely stored within the source control repository. Leveraging the existing pipeline template available in Azure Pipelines, multiple test suites are seamlessly executed. These pipelines are designed to run on a daily and weekly basis, ensuring consistent and comprehensive testing coverage. To standardize results for compatibility with NUnit, a customized tool has been developed. Azure Pipelines effectively interprets and publishes these results. Furthermore, metadata associated with the test outcomes is generated and transferred to blob storage. Once all results are aggregated within the Synapse workspace, they undergo processing, ultimately culminating in the creation of a detailed report using Power BI.
Results
The capability to export test results facilitates the creation of tailored dashboards that cater to specific requirements. Power BI emerges as a particularly valuable tool in this context, facilitating the generation of comprehensive reports that provide insights into various errors encountered during the test execution.
For each identified error, a corresponding work item is generated to expedite resolution efforts. Observations encompass a spectrum of errors, ranging from issues pertaining to client-side connections to server-side anomalies. The meticulous review of these reports forms an integral part of the sign-off process, playing a pivotal role in addressing product issues and consequently reducing overall server-side errors. While the primary focus remains on result analysis, ancillary processes such as pipeline execution, test case creation, test execution, and result publication are fully automated.
Conclusion
The utilization of Azure Load Testing for testing Microsoft Fabric and Azure Synapse has significantly bolstered the reliability of these services, establishing a resilient framework for ongoing enhancements. By diligently reviewing results on a daily basis and embracing automation wherever feasible, operational efficiency is maximized. This meticulous approach underscores the objective of subjecting the system to rigorous stress testing to ensure its capability to handle anticipated loads, thereby positioning Microsoft at the forefront of technology adoption.
