A first-hand experience of using Azure Data Factory

What is Azure Data Factory?

Pipelines

Activities

Datasets

Linked services

Moving Data

  • Reads data from source and determines a source schema
  • Uses column mapping on the target datastore to match fields from the source dataset to the target schema
  • Writes the data to sink

Data Flows

Pricing

  • Read/write and monitoring operations
  • Pipeline orchestration and execution
  • Data flow debugging and execution

Read/write and monitoring operations

Pipeline orchestration and execution

Data Flow Debugging and Execution

  • Compute optimised: $0.199 / vCore-hour
  • General purpose: $0.268 / vCore-hour
  • Memory Optimised: $0.345 / vCore-hour
  • Compute optimised: $1.582 / hour
  • General purpose: $2.144 / hour
  • Memory optimised: $2.760 / hour
  • Compute optimised: $54.128 / hour
  • General purpose: $72.896 / hour
  • Memory optimised: $93.840 / hour

Drag/Drop Interface

  • When starting a data flow debug session we’d sometimes experience an infinite loop of dialog models;
  • Clicking into a forEach activity would only load 70% of the time;
  • (this one you’ll love) if we were working off a git branch that wasn’t master and it contained different config in our linked service for our SQL database to the one that’s on the master branch, the linked service would only read/use the linked service connection defined on the master branch which made working with different dev/staging databases a nightmare.

Conclusion

--

--

--

UK-based Technology Consultant working in London and Manchester.

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

If English is your first language, you’re probably doing BEM wrong

Become an Oracle Certified Java SE 11 Developer

Development and Deployment on AWS Series Part 2: Connecting to a RDS instance using Fargate run…

Benefit of AWS- EKS

Testing responsibility shortfalls within an engineering team

Measure your website’s performance with Google Lighthouse and Protractor

GraphQL schema design: the viewer field

Node.js hosting with Heroku

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Matthew Leak

Matthew Leak

UK-based Technology Consultant working in London and Manchester.

More from Medium

DYNAMIC SQL AND EMBEDDED SQL

Usage Clause in COBOL Data Division

How to read data stored in RAM?

How to add json file inside the MS SQL table