# DataLoader.js Overview
`js/core/DataLoader.js` provides a reusable table component for displaying and editing records retrieved through `LocalSyncManager`. The same implementation is bundled for the browser in `FrontEnd/js/core/DataLoader.js`. DataLoader is designed for offline-friendly modules where data is cached in IndexedDB and later synced to the API.
## Key Features
- **Configurable Columns** – pass `columnsToShow` and `columnsToHide` to control what fields appear. The utility automatically creates table headers from object keys.
- **Global and Column Search** – an optional global search box filters all fields. Each column also receives its own search input or select box based on `searchInputTypes`.
- **Pagination** – records are fetched once and paginated client side. Page controls are rendered below the table.
- **Add/Edit Modals** – built in modals use `FormUtils` to generate form fields from `searchInputTypes`. Records are saved through `LocalSyncManager` and the shared `Toast` service notifies success.
- **Custom Edit Form** – specify `editFormId` to load a form definition from the API when editing a record. When used with `ModalBuilder`, include this property inside the table's `options` block.
- **Inline Editing** – enable `inlineEdit: true` so double-clicking a row swaps it for an inline form. When this flag is set, any `editFormId` is ignored.
```javascript
new DataLoader({ inlineEdit: true, tableName: "employees", selector: "#table" });
```
- **Delete Confirmation** – each row shows a delete button that calls
`confirm('Are you sure...')` and then `syncManager.delete`. You can provide a
custom `deleteFunction` to override this behavior.
- **Nested Tables** – `nestedTableConfig` allows a row to expand into a child table. Child rows can be added, edited and deleted while maintaining the parent linkage.
- **Row Formatting and Actions** – provide `rowFormatter` and `customRowActions` callbacks to inject custom HTML or buttons per row.
- **Row Click Handlers** – specify `rowClick` to run custom logic when a row is clicked. If omitted or invalid, the clicked row is automatically highlighted using the `selected` class.
- **Multi-Row Selection** – set `multipleSelect="true"` on the table element to allow toggling multiple highlighted rows.
- **API Row Actions** – use `apiRowActions` to attach buttons that POST/PATCH to an API endpoint for the clicked row.
- **Row Actions** – `rowActions` defines custom buttons that open forms, trigger workflows or run a callback for the clicked row.
- **Column Buttons** – show conditional buttons in any column (not just actions) with value-based rendering, dynamic styling, and API integration. See [DataLoaderColumnButtons.md](DataLoaderColumnButtons.md) for full details.
- **Offline Filter Persistence** – column filter selections are stored in `localStorage` so the view is restored when working offline.
- **Expandable Fields** – provide `expandableFields` to show a preview with a toggle for object or array values.
## Configuration Options
```javascript
new DataLoader({
selector: '#table', // required table element
tableName: 'employees', // table to query via syncManager
// optional custom API path when syncManager is auto-created
tableUrl: '../api/employees',
// optional instance of LocalSyncManager. If omitted a
// manager is created automatically using the table name
syncManager: myManager,
// restrict results by logged in user
scope: 'user',
userField: 'owner_id',
searchInputTypes: { // field => input type definitions
name: 'text',
status: { type: 'select', options: ['active','inactive'] },
review_date: { type: 'date' },
created_range: { type: 'dateRange' } // renders from/to inputs
},
columnsToShow: ['name', 'status', 'owner_id'],
sortBy: 'name',
sortDir: 'ASC',
inlineEdit: true,
// optional form id for editing existing records or use within
// a ModalBuilder table's options block
editFormId: 5,
// optional override for row deletion
deleteFunction: myDeleteHandler,
nestedTableConfig: {
childTableName: 'employee_roles',
linkField: 'employee_id',
childFormFields: { role: 'text' }
},
// columns whose values should show expandable JSON previews
expandableFields: ['details'],
// optional custom key for persisting column filters
filterStorageKey: 'employees_filters'
// function or global name called when a row is clicked
rowClick: rec => console.log(rec),
// optional transform function for each record
recordMap: rec => ({ ...rec, status: rec.isDeleted ? 'archived' : 'active' }),
// custom columns calculated from each record
calculatedColumns: { total: 'hours * rate' },
// conditional buttons in any column (see DataLoaderColumnButtons.md)
columnButtons: {
status: {
buttonMap: {
'pending': { text: 'Approve', className: 'btn-success' },
'approved': { text: 'View', className: 'btn-info' }
}
}
}
});
```
When using the `dateRange` input type, DataLoader renders two `` fields for the column: one for the starting date and one for the ending date. Selecting values stores a nested object in the `filters` map:
```json
{
"created_range": { "from": "2024-01-01", "to": "2024-01-31" }
}
```
Records are filtered inclusively so `from` uses `>=` and `to` uses `<=`. These range objects are persisted to `localStorage` alongside other filters.
Setting `scope` to `user` (or `admin_user` for admins) automatically limits records
to the logged in user by applying the specified `userField`. The same filter is
added to API requests when the sync manager is auto-created.
When using ModalBuilder:
```json
{ "type": "table", "options": { "tableName": "employees", "editFormId": 5 } }
```
### Default Hidden Fields
DataLoader automatically hides a few internal columns when rendering a new table.
These fields are considered infrastructure metadata and are excluded across **all** modules by default:
- `id`
- `createdAt`
- `parent_id`
- `isDeleted`
- `updatedAt`
- `pendingSync`
- `tenant_id`
These names are matched case-insensitively so `isDeleted`, `isdeleted`,
`updatedAt` and `updatedat` are all treated the same. Any future globally
hidden fields will also honor case-insensitive matching.
When a record includes `pendingSync: true`, the actions column displays
`syncing...` instead of the usual edit/delete buttons until the flag is
cleared.
Any column whose name contains `_id` also starts hidden unless you explicitly
include it in `columnsToShow`. The column visibility menu can still reveal these
ID fields when required.
You can still hide additional fields with `columnsToHide` or explicitly show any
of them by including the key in `columnsToShow`. They may also be re-enabled
through the column visibility menu which persists changes in the
`table_column_settings` table (see
[ColumnVisibility.md](ColumnVisibility.md) for details).
If no `syncManager` is supplied, DataLoader creates its own instance of
`LocalSyncManager` using `tableName` (and `tableUrl` when provided) so the table
can be loaded without additional setup.
The manager accepts `maxTablesPerRequest` and `maxRecordsPerRequest` options to
control how many tables or records are synced in a single request when using the
bulk API endpoints. Adjust these limits when working with large datasets or slow
connections.
### Column Visibility
Right‑click any table header to open the column visibility menu. Selections are
persisted through the `/api/table_column_settings` endpoints and follow the
inheritance order described in
[ColumnVisibility.md](ColumnVisibility.md). Columns can be dragged within that
menu to set their order. DataLoader loads the sequence from the `column_order`
field and saves changes back via the same API.
`LocalSyncManager` also exposes an `immediateGet(tableName, params)` helper. This
method builds a query string from the supplied parameters and fetches the table
data directly from the API, bypassing IndexedDB. Use it when the freshest data
is required without waiting for the next background sync.
See the source code for additional options such as `modalCustomElements`, `addButton`, and hooks for overriding default CRUD actions.
### Object and Array Fields
By default DataLoader converts object and array values into a comma‑separated
string using `formatCellValue()`. Simple arrays like `[1, 2, 3]` display as
`1, 2, 3` while objects show only their values in the same comma‑separated
format.
```javascript
new DataLoader({
tableName: 'employees'
// object and array columns render as comma separated text
});
```
To reveal full JSON details you can specify an `expandableFields` array. Each
matching column will show a preview with a toggle button. Arrays of objects are
rendered as a nested table, otherwise the raw JSON is pretty printed in a
`
` block.
```javascript
new DataLoader({
tableName: 'employees',
expandableFields: ['details', 'history']
});
```
#### Array Field Views
Some tables include array‑typed columns that contain child records (for example,
`user_emails` or `phones`). You can now control **which fields are rendered from
each child object** without trimming the data returned by the API. Define an
`arrayFieldView` on a specific column (inside the entry in `columns`) or supply a
top‑level `arrayFieldViews` map inside the table options. DataLoader will format
each child object using the provided `fields` list or `template`, while the full
object remains available for editing.
```jsonc
{
"type": "table",
"options": {
"tableName": "employees",
"arrayFieldViews": {
"user_emails": ["email"],
"phones": {
"fields": ["phone_number"],
"itemSeparator": " ",
"separator": " | "
}
}
}
}
```
- `fields` – ordered list of child properties to display. Missing or empty
values are skipped unless `keepEmpty` is `true`.
- `template` – optional string template using `{{token}}` tokens to format each
child object (overrides `fields`).
- `itemSeparator` – separator between multiple fields from the same child
object. Defaults to `", "`.
- `separator` – separator between child objects. Defaults to `", "`.
- `emptyValue` – text shown when the array is empty.
You can mix and match the two approaches by assigning `arrayFieldView` directly
on a column definition:
```javascript
columns: [
{
field: 'user_emails',
label: 'Emails',
arrayFieldView: {
template: '{{email}} ({{type}})'
}
}
]
```
The same configuration applies to the frontend bundle in `FrontEnd/js/core/DataLoader.js`,
so modules authored through ModalBuilder or handwritten views can share the
formatting rules.
### Row Selection
When a row is clicked and no valid `rowClick` handler is provided, DataLoader highlights the row by toggling the `selected` class. Add `multipleSelect="true"` to the table element to allow several rows to remain selected.
## Usage Flow
1. **Initialization** – DataLoader fetches records from `syncManager.fetchAll` and renders the first page.
2. **Filtering and Sorting** – typing in search inputs updates `filters` and re-fetches data. Sorting is applied client side.
3. **Editing Records** – clicking the edit button opens a modal. The form is serialized and sent to `syncManager.edit`.
4. **Adding Records** – the Add New button shows a modal with fields defined in `searchInputTypes`. Data is sent to `syncManager.add`.
5. **Nested Data** – expand buttons call `expandChildTable` which builds a child table filtered by `linkField` and supports its own add/edit/delete actions.
When an existing row is edited, `LocalSyncManager.edit` compares the new values
to the cached version. Only the changed columns are queued in `pendingFields`
and the row is marked with `pendingAction = 'PATCH'`. During the next bulk sync
the manager sends `{ table, id, fields }` instead of the full record. Once the
server acknowledges the update these pending flags are cleared.
### API Row Actions
Specify `apiRowActions` as an array of objects to automatically post to an API
endpoint when a row button is clicked. Each action supports:
- `buttonText` – label for the button
- `apiUrl` – endpoint template. Use `{id}` to insert the row ID.
- `method` – HTTP verb (defaults to `POST`)
- `refreshAfter` – set to `false` to skip automatic table reload
Example:
```json
{
"type": "table",
"options": {
"tableName": "vehicle_reports",
"apiRowActions": [
{ "buttonText": "Accept", "apiUrl": "../api/vehicle_reports/{id}/assign" }
]
}
}
```
### Row Actions
`rowActions` works similarly but each action can open a form, trigger a workflow or run custom logic.
- `label` – button text
- `formId`/`formName` – resolve the form and open it in a modal
- `workflow` – name of the workflow event
- `onClick` – function or global function name
```json
{
"type": "table",
"options": {
"tableName": "employees",
"rowActions": [
{ "label": "Details", "formId": 10 },
{ "label": "Activate", "workflow": "employee.activate" }
]
}
}
```
### Table View Columns (Legacy)
Table views are deprecated but older modules may still include
`table_views.columns`. Each entry may be a simple field name or an object with
additional metadata:
- `field` – database column name (required)
- `label` – custom header label
- `default` – value inserted when the record lacks this field
- `formula` – JavaScript expression evaluated using the row values
Example:
```json
[
"title",
{ "field": "assignee_id", "label": "Assignee", "default": "Unassigned" },
{ "field": "total", "formula": "hours * rate" }
]
```
Pass this array as the `columns` option when instantiating `DataLoader`. Formula strings
are compiled with `new Function('row', \`with(row){ return ${formula}; }\`)` and executed
for each record.
DataLoader applies `default` values before calculating formulas and uses
`label` when rendering the column header.
A record may also supply a `formula` property alongside the value for a
calculated column. For example, a row can return
`{ "calculated_column": 1, "formula": "likelihood * impact" }`. DataLoader
evaluates the formula when loading the record, allowing the calculation to
vary on a per-row basis when provided by the API.
### Metadata Handling
DataLoader expects the API to include a `metadata` block whenever the underlying model defines one. The bulk sync endpoint (`/api/sync/pull`) now returns `{ data, metadata }` for each requested table. `LocalSyncManager` caches this metadata so DataLoader can reuse it without extra requests. Use the helper `getMetadata(tableName)` on the sync manager to access the cached values. The metadata describes boolean fields, relations and calculated columns. DataLoader automatically processes these values to render checkbox icons, populate relation dropdowns and evaluate calculated column formulas. See [docs/ModelMetadata.md](../ModelMetadata.md) for details.
### Auto refresh on sync
`LocalSyncManager` dispatches a `tableSynced` event when a background sync
completes. DataLoader listens for this event and automatically re-fetches the
records for the relevant table so open views stay up to date. Call
`destroy()` on the DataLoader instance when unmounting to remove the listener.
## File Location
This documentation lives at `docs/frontend/DataLoader.md`. Refer here when implementing modules that use DataLoader or when modifying the class.
## Sync Manager Singleton
Recent modules use a shared helper located at `js/core/SyncManagerSingleton.js` (imported as `core/syncmanagersingleton`). Call `getManager(tableName)` to receive a single `LocalSyncManager` instance for that table. The first invocation creates and initializes the manager; later calls return the same object.
Using this singleton is important because creating multiple managers for the same table or calling `init()` repeatedly will start multiple sync loops. That can lead to duplicate rows in IndexedDB when background sync runs.
Dynamic modules like **Vehicles** obtain their managers from `getManager()` and you should follow the same pattern whenever you need a manager outside of DataLoader.
## Auto Push and `pushRecord()`
`LocalSyncManager` accepts an optional `autoPush` boolean. When true any call to
`add()` or `edit()` immediately pushes the affected record to the server using
the bulk sync endpoint. The same behaviour can be triggered on a per-call basis
via the `pushNow` option:
```javascript
syncManager.add('employees', data, { pushNow: true });
```
Additionally, the manager exposes `pushRecord(tableName, recordOrId)` to send a
single record without waiting for the next background sync. Provide either the
record object or a record ID. Temporary IDs returned from the API replace the
local value and pending flags such as `pendingSync` are cleared automatically.
When using WebSockets, push acknowledgements arrive as a `pushResult` message
which updates the local store in the same way.
## Auto Sync
`autoSync` triggers a pull after a record is pushed. When enabled and the
browser is online, `add()`, `edit()` and `delete()` call
`immediateSync(tableName)` once `pushRecord()` finishes. You can force this for a
single operation using the `syncNow` option:
```javascript
syncManager.edit('employees', data, { syncNow: true });
```
The sync step is skipped if the browser is offline.
When a WebSocket connection is active via `startWebSocketListener`, `syncNow()`
tries to push pending records over that socket before using the bulk endpoint.
This keeps small updates real-time while still falling back to `/bulk/push` when
the connection is unavailable.
WebSocket connections are coordinated by `WebSocketManager`. The first manager
to call `startWebSocketListener` creates the socket and sends the subscription
payload. Additional managers call `startWebSocketListener` as well, but they
reuse the shared socket and simply register their message handler.
`add()` now follows the same approach. If the WebSocket is open the new record
is sent using `sendUpdate()` and HTTP sync is skipped.
## Partial Row Updates
`LocalSyncManager.startWebSocketListener()` emits a `tableUpdated` event for each
record received over the WebSocket. The event detail now includes
`{ tableName, action, record }` so `DataLoader` can update rows without fetching
the entire table. When `action` and `record` are present, the loader updates its
`localData` array and re-renders only the affected row. If those details are
missing the loader falls back to `fetchData()` which refreshes all rows.