Implement Short Polling in Azure Data Factory

When building data workflows, you might need to poll a RESTful API at regular intervals until a specific condition is met. Azure Data Factory (ADF) makes it easy to implement this using Until, Web, and Wait activities. In this article, we will walk through setting up short polling in ADF. Let’s get started!

Steps

1. From Azure Data Factory’s Author option, create a new pipeline.
2. Inside that pipeline, select Until and drag it over the editor canvas.

3. Inside Until click on the plus sign (+) and select Web.

4. After the Web, click the plus sign (+) and select Wait.

5. Now lets go inside Until1 Activity and click on the Web1 activity. Then select Settings to configure the url and parameters.

6. Then, click on Wait1 and set up the waiting time for short poll, because you do not want to poll the url without an interval.

7. Lets go back to Until1 Activity (the parent activity of Web1 and Wait1) and select Settings. Click on Expression.

8. Select Dynamic Expression and set up the appropriate end condition for Until1 Activity.

9. Now save everything and try to run it by using Debug. The Output should look like this:

10. Thats it, your short polling should work now until you get your desired output from the Rest API.

In Conclusion

This approach ensures that you don’t overwhelm your API with constant requests, while still allowing your pipeline to react and process data when the desired outcome is achieved. By adjusting the waiting time and setting dynamic expressions, you can customize the polling frequency and end conditions to suit your specific use case.