Advanced extraction
Advanced data extraction, when there's no clean way in
No API, no documentation, an export that drops half the columns. Extraction is where most migrations, integrations and warehouses stall, and it is the part we specialise in.
- 01Undocumented systemNo API, no export worth the name
- 02Traced and parsedRead-only, every read logged
- 03Tables we can queryCounted and reconciled
Techniques
- 01
Watch the database while you use the app
Do one thing in the application and trace which tables change. Repeat until the undocumented schema maps itself. It is the fastest way to learn what a status code or a flag really means.
- 02
Infer keys and relationships from the data
With no foreign keys declared, value overlap, cardinality and naming patterns show which columns join to which. Every inferred link is tested before anything is built on it.
- 03
Read the file format directly
Workbooks, embedded databases and proprietary files can be parsed without the application that made them. That is how we read 113 VBA modules straight out of a workbook. Read the write-up
- 04
Use the application's own back end
Modern interfaces fetch their data from internal endpoints. Captured once, those calls are often a better API than the official one.
- 05
Parse what the reports print
Fixed-width listings, report files and exports, turned into tables and checked line by line against the report's own control totals.
- 06
Extract only what changed
Watermarks, row versions and hashes, so a large source can be pulled in increments and re-pulled safely after a failure.
- 07
Pull structure out of free text
Phone numbers, emails, dates and intent buried in notes fields, moved into proper columns by rules you can read and check.
- 08
Automate the screens
The last resort, and sometimes the only one: browser automation that reads each record slowly, logs everything and resumes where it stopped.
Whatever the route
Every extraction ends in tables we can query and count, with a manifest of what was read and when, reconciled against what the system itself reports. If the two disagree, you hear about it before anything is built on the data.
Sources
- SQL Server, MySQL, PostgreSQL, Access and anything with an ODBC or JDBC driver.
- REST, SOAP and OData APIs, including the undocumented ones behind a web interface.
- Excel workbooks and the VBA inside them.
- SharePoint lists and document libraries.
- CSV, fixed-width and XML files, report outputs and SFTP drops.
- Web applications with no export at all.
Permission
We extract only from systems you own or are licensed to use, with written permission from the data owner, and within the vendor's terms where they apply. Access is read-only and scoped to what the job needs.
Got a system that won't let go?
Tell us what it is, what you can log in to, and what you need out of it.