Prerequisites
Before configuring the source, your office needs the following from XP. They are all obtained through the XP developer portal and Security Assessment:- Client ID and Client Secret: OAuth2 credentials generated in the developer portal (Gestão → Credenciais). The client secret is shown only once.
- Subscription Key: sent by XP by email after the Security Assessment is approved (the
Ocp-Apim-Subscription-Keyvalue). - Digital certificate: a client-authentication certificate (mTLS) issued with your office’s data. It must have the Client Authentication extended key usage (
1.3.6.1.5.5.7.3.2) and OCSP enabled, and it must not include the TLS Web Server Authentication usage. ICP-Brasil certificates are accepted. - IP allow-listing: XP releases access by IP range. The IPs that reach the API are registered in the Security Assessment ticket.
Configuring XP as a Source
In the Sources tab, click on the “Add source” button located on the top right of your screen. Then, select the XP option from the list of connectors. Click Next and you’ll be prompted to add your access.1. Add account access
Provide the credentials and certificate that XP issued to your office:- Environment: The XP Data Access environment to connect to. Choose
homologationto validate the integration first, orproductionfor live data. Each environment has its own credentials, certificate, and scope. - Client ID: The OAuth2 client ID from the developer portal.
- Client Secret: The OAuth2 client secret from the developer portal.
- Subscription Key: The
Ocp-Apim-Subscription-Keyvalue sent by XP. - Certificate file (.pfx): The digital certificate file in PKCS#12 format (
.pfx/.p12), uploaded exactly as delivered by the certificate authority. No conversion needed. - Certificate Password: The password of the
.pfxcertificate file. Leave empty only if the certificate has no password. - Partner Name: An identifier for your office, used in the request
User-Agent. - Start Date (advanced): The earliest record date to sync for incremental streams.
- Customer Codes (advanced): Restricts the per-customer streams (balances, consolidated position, investment account statement, positions by date and operations) to the customers you list. Leave it empty to read every account whose registry row is current. Each customer costs at least one API request per per-customer stream, so this is how you pilot or cap a source that runs several times a day.
- Per-customer history start date (advanced): The first day read by the investment account statement, positions by date and operations streams on their first sync and on a full sync. Leave it empty to start 15 days back. It only matters before a stream’s first run: changing it later does not reload older days. Each 15-day period costs one API request per customer per stream, so a long history is loaded over several runs.
- Max windows per run (advanced): How many 15-day periods the investment account statement, positions by date and operations streams read in each run (default 2). A daily run needs only one; a longer history is loaded a few periods per run, each run continuing where the previous one stopped.
- Excluded customer codes (advanced): Customers that the consolidated position, investment account statement, positions by date and operations streams never request. Use it for a customer XP rejects on every run: the run’s error message lists those customers. Write each code exactly as XP returns it and as the error lists it, since codes are compared as text (
001234does not match1234). The balance streams ignore it. - Operations product types (advanced): The product types requested by the operations stream. Keep the default unless XP rejects it.
.pfx file, use the advanced settings: Client Certificate (PEM) and Private Key (PEM) accept the certificate and key in PEM format (leave the private key empty if the certificate already includes it). The uploaded .pfx file takes precedence when both are provided.
Once you’re done, click Next.
2. Select streams
Choose which data streams you want to sync. For faster extractions, select only the streams that are relevant to your analysis. You can select entire groups of streams or pick specific ones.Tip: The stream can be found more easily by typing its name.Select the streams and click Next.
3. Configure data streams
Customize how you want your data to appear in your catalog. Select the desired layer where the data will be placed, a folder to organize it inside the layer, a name for each table (which will effectively contain the fetched data) and the type of sync.- Layer: choose between the existing layers on your catalog. This is where you will find your new extracted tables as the extraction runs successfully.
- Folder: a folder can be created inside the selected layer to group all tables being created from this new data source.
- Table name: we suggest a name, but feel free to customize it. You have the option to add a prefix to all tables at once and make this process faster!
- Sync Type: you can choose between INCREMENTAL and FULL_TABLE.
- Incremental: every time the extraction happens, we’ll get only the new data, which is good if, for example, you want to keep every record ever fetched.
- Full table: every time the extraction happens, we’ll get the current state of the data, which is good if, for example, you don’t want to have deleted data in your catalog.
4. Configure data source
Describe your data source for easy identification within your organization, not exceeding 140 characters. To define your Trigger, consider how often you want data to be extracted from this source. This decision usually depends on how frequently you need the new table data updated (every day, once a week, or only at specific times).- Configure Delta Log Retention and determine for how long we should store old states of this table as it gets updated. Read more about this resource here.
- Determine when to execute an Additional Full Sync. This will complement the incremental data extractions, ensuring that your data is completely synchronized with your source every once in a while.
5. Check your new source
You can view your new source on the Sources page. If needed, manually trigger the source extraction by clicking on the arrow button. Once executed, your data will appear in your Catalog.Streams and Fields
Below you’ll find all available data streams from XP and their corresponding fields. Most streams sync incrementally: dimensions bylastUpdate, fact tables by dimTimeCode (a YYYYMMDD time key), and Positivador by positionDate. They carry data as of D-1 (the previous day) and are updated only on B3 business days.
The balance streams work differently: they read the customer’s balance as it stands at the moment of the sync (D0). XP exposes them one customer at a time, so the source makes one API request per customer on every run, and the balance is stored as a snapshot: each run refreshes the customer’s row instead of appending history, with extractedAt recording when the snapshot was taken. Because XP allows 300 requests per minute per subscription across all of its APIs, how long a run takes grows with the size of your customer base; use the Customer Codes advanced setting to restrict a source to a subset of customers.
Four more per-customer streams keep history instead of a single snapshot: Consolidated Position, Investment Account Statement, Positions by Date and Operations. They are not selected by default, because each one adds at least one API request per customer to every run: with about 6,000 customers, that is roughly 20 to 25 minutes per stream per run. Consolidated Position and the statement are current-day data (D0). Positions by Date and Operations are published by XP three business days later (D-3), so running them more than once a day brings no new data.
Account (Conta)
Account (Conta)
currentRegisterIndicator marks the active row.Key Fields:id- Unique record identifierdimAccountCode- Surrogate key for the account versionaccountCode- Stable business key of the accountcpfCnpjCodeGuid- Anonymized CPF/CNPJ identifierpersonType- Person type (F for individual, J for company)dscSuitability- Suitability profile descriptionstartValidityDate/endValidityDate- Validity window of this versioncurrentRegisterIndicator- 1 for the current row, 0 otherwiselastUpdate- Last update timestamp (replication key)
Advisor (Assessor)
Advisor (Assessor)
account_advisor_relation joins on dimAdvisorCode.Key Fields:id- Unique record identifierdimAdvisorCode- Surrogate key for the advisor versionadvisorCode- Stable business key of the advisoradvisorName/advisorEmail/username/cpfCode- Advisor identificationsituation- Advisor situation/statusmaster- Master (parent) advisor identifierxpEntryDate/xpExitDate- Dates the advisor joined/left XPvalidityStartDate/validityEndDate- Validity window of this versioncurrentRegister- Whether this is the current rowlastUpdate- Last update timestamp (replication key)
Office Channel (Canal do Escritório)
Office Channel (Canal do Escritório)
id- Unique record identifierdimOfficeChannelCode- Office channel surrogate keyofficeCode/officeName/officeFantasyName- Office identificationcity/state/neighborhood- LocationmasterAdvisorCode- Master advisor codeheadOfficeCode/headOfficeName- Head officevalidityStartDate/validityEndDate- Validity windowcurrentRegister- 1 for the current row, 0 otherwiselastUpdate- Last update timestamp (replication key)
Product (Produto)
Product (Produto)
id- Unique record identifierdimProductCode- Product surrogate keyassetCode/assetName- Asset identificationproductClassficationL0toproductClassficationL5- Classification levels (key spelling matches the API)isinCode/cnpjCode/cetselCode- External identifiersissuerName/managerName/strategy- Issuer and managementproductType- Product type (DB, LCI, LCA, LC, LF, LIG, and others)yield/index/dealType- Yield attributeslastUpdate- Last update timestamp (replication key)
Movement Type (Tipo de Movimentação)
Movement Type (Tipo de Movimentação)
id- Unique record identifierdimMovementTypeCode- Movement type surrogate keymovementTypeCode- Movement type codemovementTypeDescription- Movement type descriptionlastUpdate- Last update timestamp (replication key)
Financial Institution (Instituição Financeira)
Financial Institution (Instituição Financeira)
id- Unique record identifierdimFinancialInstitutionCode- Financial institution surrogate keyfinancialInstitutionCode/financialInstitutionName- Institution identificationcategoryName- CategorylastUpdate- Last update timestamp (replication key)
Time (Tempo)
Time (Tempo)
id- Unique record identifierdimTimeCode- Time key (YYYYMMDD)date- Calendar dateyearNumber/monthNumber/dayNumber- Date partsquarterNumber/weekOfTheYearNumber- Period attributesbusinessDay/tradingDay- 1 if it is a business/trading day, 0 otherwisenextTradingDate- Next trading date
Media (Midia)
Media (Midia)
additionalFields carries technical metadata as a JSON string.Key Fields:id- Unique record identifiercodConta- Account codemarca- Branddevice- DevicebaseUrl- Base URLadditionalFields- Additional technical metadata (JSON string)lastUpdate- Last update timestamp (replication key)
Account-Advisor Relation (Relação Conta-Assessor)
Account-Advisor Relation (Relação Conta-Assessor)
id- Unique record identifierdimAccountCode/accountCode- Account referencesdimAdvisorCode/advisorCode- Advisor referencesdimOfficeChannelCode- Office channel referencedate- Relation reference datedimTimeCode- Time key (YYYYMMDD, replication key)lastUpdate- Last update timestamp
Inflow (Captação)
Inflow (Captação)
id- Unique record identifiermovementCode- Movement codemovementNatureCode- Movement naturemovementAmount/movementValue- Quantity and valuedimAccountCode/dimProductCode/dimAdvisorCode- Dimension referencesdimFinancialInstitutionalCode- Financial institution referencedimMovementTypeCode- Movement type referencedimTimeCode- Time key (YYYYMMDD, replication key)lastUpdate- Last update timestamp
Custody (Custódia / AUC)
Custody (Custódia / AUC)
id- Unique record identifierdimAccountCode/dimProductCode- Account and product referencesdimOfficeChannelCode/dimAdvisorCode- Office and advisor referencespositionAmount- Position quantitypositionValue- Position value (assets under custody)termDueDate- Term due datedimTimeCode- Time key (YYYYMMDD, replication key)lastUpdate- Last update timestamp
Active Customer (Cliente Ativo)
Active Customer (Cliente Ativo)
id- Unique record identifierdimAccountCode- Account referenceaccountQuantity- Number of accountsdimOfficeChannelCode/dimAdvisorCode- Office and advisor referencesoperatedStockExchange- Whether the client traded equities in the last 30 daysdimTimeCode- Time key (YYYYMMDD, replication key)lastUpdate- Last update timestamp
Positivador
Positivador
id- Unique record identifieraccountCode/advisorCode/headOfficeCode- Identificationsegment/segmentClient/dscSuitability- Client profilestatus/activatedInM/churnedInM- LifecyclefinancialApplications/revenueInMonth- Balances and revenuegrossCaptureInMonth/redemptionInMonth/netCaptureInMonth- Capture metricsnetInMonthand relatednet*fields: Net positions by product classpositionDate- Position date (replication key)
Reprocessing Log (Log de Reprocessamento)
Reprocessing Log (Log de Reprocessamento)
maximumProcessingDate to know how current each resource is.Key Fields:tableName- Name of the reprocessed resourcereferenceDate- Reference date of the datatypeProcessing- Processing type (FULL or INCREMENTAL)minimumProcessingDate/maximumProcessingDate- Processing window
Digital Account Balance (Saldo Conta Digital)
Digital Account Balance (Saldo Conta Digital)
customerCode- XP code of the customer the balance belongs to (primary key)agency- Identifier of the digital account’s branchtradingAccount- Identifier of the digital accountamount- Amount available for transactionsblockedAmount- Amount blocked and unavailable for useamountWithOverdraft- Amount available including the overdraft limitoverdraftLimit/overdraftAvailable/overdraftConsumed- Overdraft granted, still available, and already usedlastChangeDate- When the balance last changed, as reported by XPextractedAt- When this snapshot was read from the API
Investment Account Balance (Saldo Conta Investimento)
Investment Account Balance (Saldo Conta Investimento)
customerCode- XP code of the customer the balance belongs to (primary key)brand- Brand of the investment accounttradingAccount- Number of the trading accountwhiteLabel- White label identifier of the accountamount- Balance available in the investment accountlastUpdate- When the balance was last updated, as reported by XPextractedAt- When this snapshot was read from the API
Consolidated Position (Posição Consolidada)
Consolidated Position (Posição Consolidada)
customerCode- XP code of the customer (primary key)positionDate- Day of the position (primary key)patrimonioTotal/patrimonioTotalLiquido/investimentos/valorDisponivel- Totalsfinanceiro*- Cash, projected balances (D+1 to D+3), collateral and pending redemptionsvalorBensMoveis/valorBensImoveis/valorAplicacoesFinanceiras/totalPatrimonioDeclarado- Wealth declared by the customerprodutos- Position in each product type (stocks, funds, fixed income, treasury, pension, COE and others) with its items, as a JSON stringdadoAtualizado/dataAtualizacao/atualizeEm- Whether XP reports the data as final, when it was last updated and when XP suggests reading it againextractedAt- When the position was read (replication key)
Investment Account Statement (Extrato Conta Investimento)
Investment Account Statement (Extrato Conta Investimento)
customerCode- XP code of the customer (primary key)entryNumber- Identifier of the entry (primary key)entryDate- Day of the entry (replication key)referenceDate/settlementDate- Reference and settlement datesvalue- Amount of the entry (negative for debits)balance- Account balance after the entrydescription/historicalCode- What the entry isisFuture- Whether the entry is scheduled for a future datebrand/tradingAccount/whiteLabel- Account the statement belongs tolastUpdate- When XP last updated the entry
Positions by Date (Posição V2)
Positions by Date (Posição V2)
customerCode- XP code of the customer (primary key)positionKey- Identifier of the position created by Nekt, since XP sends none (primary key)category- Product typepositionDate- Day of the position (replication key)assetId/asset/isin/fundCNPJ/cetipSelicCode- Asset identificationopeningQuantity/closingQuantity/openingValue/closingValue/closingNetValue- Quantities and values at the opening and close of the dayyield/previousYield/accumulatedYield/profitAndLoss- Return of the positionincomeTax/iof- Taxesdetails- Every field XP sent for the position, as a JSON string
Operations (Movimentações de Operações)
Operations (Movimentações de Operações)
customerCode- XP code of the customer (primary key)operationKey- Identifier of the operation created by Nekt, since XP sends none (primary key)category- Product typeeventDate- Day of the operation (replication key)asset/assetId/isin/fundCNPJ- Asset identificationtype/eventType/operationType- Kind of operationquantity/unitPrice/grossValue/netValue- Quantity and valuesincomeTax/iof- TaxessettlementDate- Settlement datedetails- Every field XP sent for the operation, as a JSON string
Implementation Notes
Authentication Retries
The XP gateway intermittently returns401 Unauthorized errors for requests that have valid credentials. To prevent syncs from failing due to this gateway instability, the connector automatically intercepts 401 responses, discards the cached OAuth token, requests a fresh one, and retries the request. A sync will only fail with an authentication error if the 401 response persists across every retry attempt, which indicates that the credentials were genuinely rejected or have expired.
Per-Customer Streams with History
- Cost and schedule: these streams read one customer at a time and share XP’s limits with every other XP source of the same subscription: 300 requests per minute, and at most 20 GB of data transferred every 12 hours. Schedule sources so their runs do not overlap, and keep Positions by Date and Operations, which XP publishes three business days late, on a daily trigger.
- How much history: the Investment Account Statement, Positions by Date and Operations streams read 15-day periods, one period at a time for all customers, and remember how far they got after each complete period. Each run starts some days before that point, to pick up late postings and corrections: 7 days for the statement, and 13 days for Positions by Date and Operations, since XP publishes them three business days late, which around the year-end holidays means a week or more. Each run reads at most Max windows per run periods. Their first run starts at Per-customer history start date, or 15 days back. Because each run starts again a few days before where the previous one stopped, the periods overlap: with the default of 2 periods per run, each run moves forward about 16 days for Positions by Date and Operations and about 23 days for the statement. One year of history is therefore loaded over about 23 runs, roughly 250,000 requests per stream for 6,000 customers (about 16 runs for the statement). A customer that joins your office later gets data from one to two weeks back on. A day that XP publishes more than about two weeks late is not read again.
- Avoid full syncs on these streams: a full sync rebuilds the table starting at Per-customer history start date (or 15 days back), reading the whole period in a single run, so older days are removed. For Consolidated Position, a full sync keeps only today’s position and removes every earlier day. The run log warns about both. Do not schedule an Additional Full Sync for them.
- Errors are not skipped: the only answer that skips a customer is XP saying the customer does not belong to your office; those customers are listed in the run log. Any other error for a customer (400 Bad Request or 404 Not Found) is data XP did not send, so the run reads the rest of the period and then fails, and the next run reads that period again. If ten customers in a row get such an error, the run stops right away, since the request itself is likely wrong. If XP rejects the same customer on every run, add it to Excluded customer codes so the other customers keep loading.