Appearance
Kibanna
Kibanna is available as a docker image inside of your config file. And it is inside the multiplexer.
Outside of the multiplexer you open it as well as a stand alone.
bun run kibanaCluster Configuration
Because some developers might require more needs in the data cluster than others, this is configurable via the config file.
ts
coreDataCluster: z.object({
instanceCount: z.number().default(2),
instanceType: z.string().default("t3.small.search"),
}).default({
instanceCount: 2,
instanceType: "t3.small.search",
}),Sync
For each index you can define a list of queries, that includes a query and a size parameter. The query is an Opensearch Query, and the size is the maximum number of documents to sync.
Each query is memorized, and so it will not rerun queries that have not changed.
💡 If you want to sync more data, don't broaden an already synced query, instead make a new query. This allows for a more holistic spread of data, and better performance.
In addition to re-syncing when queries have changed or been added, it will listen to updates from the parent data cluster (mi-playground). When new versions of data from the parent cluster come out, it will re-sync those indexes. This is a really useful dogfood of our data pipeline.
Configuration
You can configure your core data sync process in your config file.
ts
/**
* CoreDataSync is the initial sync of data from the core data
* cluster with all 60 million plus rows of data, with your local
* instance.
*
* You can provide an array of queries to run, they will all be run
* up to the size you provide.
*
* CoreDataSync can also be set to false, if you do not want to
* sync data at all for a stage. Like "main" or "-live" stages that
* share data with their "dev" counterparts.
*
* The data will not run again unless you change the config.
*
* @example
* ```json
* {
* "coreDataSync": {
* "loans": {
* "queries": [
* {
* "query": {
* "bool": {
* "must": [
* {
* "term": {
* "mm_address.city.keyword": "long beach"
* }
* }
* ]
* }
* }
* }
* ]
* }
* }
* }
* ```
*/
coreDataSync: z
.object({
/**
* The queries to run for the loans core data sync.
*
* This comes from everstorm_loans index.
*/
loans: z
.object({
/**
* The queries to run for the loans core data sync.
*/
queries: z.array(
z.object({
/**
* The query to run for the loans core data sync.
*/
query: z
.any()
.transform((val) => val as unknown as QueryContainer),
/**
* The total size to pull, unless less are present.
*/
size: z.number(),
}),
),
})
.optional(),
/**
* The queries to run for the properties core data sync.
*
* This comes from everstorm_properties index.
*/
properties: z
.object({
/**
* The queries to run for the properties core data sync.
*/
queries: z.array(
z.object({
/**
* The query to run for the properties core data sync.
*/
query: z
.any()
.transform((val) => val as unknown as QueryContainer),
/**
* The total size to pull, unless less are present.
*/
size: z.number(),
}),
),
})
.optional(),
/**
* The queries to run for the originators core data sync.
*
* This comes from everstorm_originators index.
*/
originators: z
.object({
/**
* The queries to run for the originators core data sync.
*/
queries: z.array(
z.object({
/**
* The query to run for the originators core data sync.
*/
query: z
.any()
.transform((val) => val as unknown as QueryContainer),
size: z.number(),
}),
),
})
.optional(),
/**
* The queries to run for the agents core data sync.
*
* This comes from everstorm_agents index.
*/
agents: z
.object({
/**
* The queries to run for the agents core data sync.
*/
queries: z.array(
z.object({
/**
* The query to run for the agents core data sync.
*/
query: z
.any()
.transform((val) => val as unknown as QueryContainer),
/**
* The total size to pull, unless less are present.
*/
size: z.number(),
}),
),
})
.optional(),
/**
* Whether to automatically start the core data sync
* in dev mode.
*
* @default true
*/
autostart: z.boolean().default(true),
})
.default({
autostart: true,
loans: {
queries: [
{
query: {
bool: {
must: [{ term: { "mm_address.city.keyword": "long beach" } }],
},
},
size: 1000,
},
],
},
properties: {
queries: [
{
query: {
bool: {
must: [{ term: { "mm_address.city.keyword": "long beach" } }],
},
},
size: 1000,
},
],
},
}),