Extraction · Migration · Integration · Warehousing
We get data out of any system, with or without an API, then migrate it, integrate it or build a warehouse on it, and reconcile every load so you can sign it off.
Based in Glossop, near Manchester. Working across the UK, on site when it helps.
Who does the work
Data engineer and architect, Glossop, near Manchester
15 years of moving, modelling and reconciling business data, mostly on the Microsoft stack, across cybersecurity, social housing, manufacturing and professional services. The person who reads your enquiry writes the code.
Ways in
The access you have decides the extraction route. Pick the closest.
The route
Services
Data out of systems with no API, no documentation or no export worth the name: traced databases, parsed file formats, internal endpoints and, when nothing else works, automated screens.
You get: Extraction code, a manifest of what was read, reconciliation against the source's own totals.
One or more old systems into a new one: extraction, mapping, transformation, a rehearsed cutover and a reconciliation you can sign.
You get: Mapping specification, migration code, rehearsal reports, cutover runbook, sign-off sheet.
Systems kept in step after go-live: scheduled syncs, API integrations and file feeds that run unattended and say clearly when a person is needed.
You get: Interface specification, integration code, monitoring and alerts, runbook.
Warehouses the business can report from: dimensional models, incremental loads and data quality checks on every run, with Power BI on top.
You get: Pipelines, warehouse model, data dictionary and lineage, Power BI models and reports.
Specialist extraction, migration and warehousing engineering for agencies, consultancies and systems integrators, working to your delivery lead.
You get: Whatever your workstream needs, to your standards.
Spreadsheets into systems
Macros, copies of copies, and one person who knows how it works. We read the workbook end to end, then rebuild it as a proper system, with its data moved across and reconciled.
The workbook
The system
Edge cases
Bespoke systems and old data produce problems nobody plans for. Finding them before the load is most of the job.
Meaning
The same column holds daily rates in one system and hourly rates in another.
Units are checked against totals before anything loads. In one migration this stopped project costs reading eight times too high.
Meaning
Users worked around a missing field by typing into a free-text one, for years.
Rules pull dates, phone numbers and emails out into real columns, and every value can be traced back to the note it came from.
Formats
UTF-8 text read as Windows-1252 somewhere along the way and saved again, so every £ gains a stray character.
We detect the encoding column by column, repair the damaged values, and count every repair on the sign-off sheet.
Access and rules
Archived items it treats as read-only, entries over its limits, values outside its picklists.
Every record is tested against the target's rules in SQL first. Then we decide with you what to fix, split or hold back.
Method
Every cutover has a freeze window. We measure ours in rehearsal and tell you the number before you commit to a date.
Read-only access to every source. Each table counted, keyed and checked for orphans, duplicates and values that don't fit their column.
You getProfiling report
Field by field, source to target, with every transformation, lookup and default written down in plain language.
You getMapping specification, signed off by you
Extraction and transformation code, with the validation rules alongside it, run inside your environment wherever possible.
You getRepeatable migration code
Full dry runs into a test copy of the target. Each one timed, reconciled and reviewed with your team.
You getRehearsal reports and a measured cutover window
Freeze, final extract, load and checks, then a go or no-go call made against criteria agreed in advance.
You getExecuted cutover runbook
Counts, totals and record-level spot checks, source against target, for every object. Anything not moved is listed with the reason.
You getSign-off sheet
Proof
Counts and totals for every object, source against target. Anything not moved is listed with its reason. Nothing is signed off on trust.
Download the reconciliation checklist| Object | Source | Loaded | Held back | Reconciled |
|---|---|---|---|---|
| Accounts | 18,204 | 18,177 | 27 | 27 soft-deleted in source, listed |
| Contacts | 41,870 | 41,870 | 0 | |
| Orders | 266,512 | 266,512 | 0 | |
| Order lines | 1,204,880 | 1,204,880 | 0 | |
| Order value (£) | 48,227,615.40 | 48,227,615.40 | 0.00 | |
| Open balance (£) | 1,903,118.22 | 1,903,118.22 | 0.00 | |
| Attachments | 9,912 | 9,908 | 4 | 4 zero-byte files, listed |
Plus record-level spot checks, the full held-back list and the mapping specification version it ran against.
Write-ups
Clients aren't named, and nor is anything that would identify them. The problems, and how they were found, are real.
8×how far too high project costs would have read, caught before load
A professional services firm moving nine object types from two project and timesheet systems into one cloud platform. Every rule was tested in SQL before anything loaded.
32,000lines of undocumented VBA, read and reviewed
A housing association needed to know what was inside a large Excel and VBA application before deciding its future. We pulled every module out with a purpose-built parser and reviewed all of it.
2 yearsleading the data team, before Bit Movers
At a global cyber security consultancy: the business-critical warehouse, the Power BI reporting on it, and integrations for expense and HR data.
Stack
When your project needs an extraction, migration or warehousing specialist for a few weeks, we join your team: your delivery lead, your standards, your client.
How partner work runs
Tell us what you are moving from and to, and how the data can be reached. No data or credentials at this stage: a description is enough.